2020-12-05T18:51:17.662 INFO:root:teuthology version: 1.0.0-e6b33b7b 2020-12-05T18:51:17.664 DEBUG:teuthology.run:Teuthology command: teuthology --lock --name teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi --owner scheduled_teuthology@teuthology --block --verbose --archive /home/teuthworker/archive/teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi/5667657 --description rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} -- /tmp/teuthology-worker.ve5_qfv1.tmp 2020-12-05T18:51:17.698 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-12-05T18:51:17.761 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi/5667657 branch: octopus description: rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: ceph-qa@ceph.io first_in_suite: false job_id: '5667657' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: octopus ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 3b1861acb8137d863184795ccadbd5d0cb026674 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 3b1861acb8137d863184795ccadbd5d0cb026674 rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: octopus sha1: 3b1861acb8137d863184795ccadbd5d0cb026674 owner: scheduled_teuthology@teuthology priority: 1000 repo: git://git.ceph.com/ceph.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 3b1861acb8137d863184795ccadbd5d0cb026674 sleep_before_teardown: 0 suite: rgw suite_branch: octopus suite_path: /home/teuthworker/src/git.ceph.com_ceph_octopus/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph.git suite_sha1: 3b1861acb8137d863184795ccadbd5d0cb026674 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2020-11-30_05:05:02 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.26405 2020-12-05T18:51:17.761 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_octopus/qa; will attempt to use it 2020-12-05T18:51:17.762 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks 2020-12-05T18:51:17.762 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-12-05T18:51:17.764 INFO:teuthology.task.internal:Checking packages... 2020-12-05T18:51:17.783 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '3b1861acb8137d863184795ccadbd5d0cb026674' 2020-12-05T18:51:17.784 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-12-05T18:51:17.784 INFO:teuthology.packaging:ref: None 2020-12-05T18:51:17.785 INFO:teuthology.packaging:tag: None 2020-12-05T18:51:17.785 INFO:teuthology.packaging:branch: octopus 2020-12-05T18:51:17.785 INFO:teuthology.packaging:sha1: 3b1861acb8137d863184795ccadbd5d0cb026674 2020-12-05T18:51:17.786 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2020-12-05T18:51:18.078 INFO:teuthology.task.internal:Found packages for ceph version 15.2.7-615-g00b4001e-1bionic 2020-12-05T18:51:18.079 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-12-05T18:51:18.090 INFO:teuthology.task.internal:no buildpackages task found 2020-12-05T18:51:18.090 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-12-05T18:51:18.105 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-12-05T18:51:18.106 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-12-05T18:51:18.156 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_teuthology@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi/5667657'} 2020-12-05T18:51:18.175 DEBUG:teuthology.lock.ops:locked ubuntu@smithi185.front.sepia.ceph.com, ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T18:51:18.222 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd20fde1a60>, signals=[15]) 2020-12-05T18:51:18.432 INFO:teuthology.provision.fog.smithi185:Scheduling deploy of ubuntu 18.04 2020-12-05T18:51:18.442 INFO:teuthology.provision.fog.smithi023:Scheduling deploy of ubuntu 18.04 2020-12-05T18:51:18.845 INFO:teuthology.orchestra.console:Power off smithi185 2020-12-05T18:51:18.846 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi185.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-12-05T18:51:18.867 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi185.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:18.879 INFO:teuthology.orchestra.console:Power off smithi023 2020-12-05T18:51:18.880 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-12-05T18:51:18.899 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:22.894 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi185.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:22.920 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:27.006 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi185.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:27.032 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:27.218 INFO:teuthology.orchestra.console:Power off for smithi185 completed 2020-12-05T18:51:27.241 INFO:teuthology.orchestra.console:Power off for smithi023 completed 2020-12-05T18:51:27.320 INFO:teuthology.orchestra.console:Power on smithi185 2020-12-05T18:51:27.320 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi185.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-12-05T18:51:27.342 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi185.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:27.349 INFO:teuthology.orchestra.console:Power on smithi023 2020-12-05T18:51:27.349 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-12-05T18:51:27.369 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:31.362 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi185.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:31.390 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:35.474 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi185.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:35.499 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-12-05T18:51:35.685 INFO:teuthology.orchestra.console:Power on for smithi185 completed 2020-12-05T18:51:35.708 INFO:teuthology.orchestra.console:Power on for smithi023 completed 2020-12-05T18:51:35.786 INFO:teuthology.provision.fog.smithi185:Waiting for deploy to finish 2020-12-05T18:51:35.809 INFO:teuthology.provision.fog.smithi023:Waiting for deploy to finish 2020-12-05T18:54:37.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:54:46.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:54:52.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:54:55.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:01.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:04.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:10.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:13.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:19.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:22.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:28.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:31.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:37.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:40.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:47.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:55:56.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:56:05.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:56:18.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:56:18.365 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-12-05T18:56:24.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:56:24.382 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-12-05T18:56:30.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:56:30.397 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-12-05T18:56:36.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:56:36.410 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-12-05T18:56:42.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:56:42.424 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-12-05T18:56:48.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:56:48.551 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-12-05T18:56:48.552 INFO:teuthology.orchestra.run.smithi185:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-12-05T18:56:48.959 INFO:teuthology.provision.fog.smithi185:Node is ready 2020-12-05T18:56:48.959 INFO:teuthology.orchestra.run.smithi185:> hostname 2020-12-05T18:56:49.005 INFO:teuthology.orchestra.run.smithi185.stdout:smithi185.front.sepia.ceph.com 2020-12-05T18:56:49.005 INFO:teuthology.orchestra.run.smithi185:> grep smithi185.front.sepia.ceph.com /etc/hosts 2020-12-05T18:56:49.053 INFO:teuthology.orchestra.run.smithi185.stdout:172.21.15.185 smithi185.front.sepia.ceph.com smithi185 2020-12-05T18:56:49.054 INFO:teuthology.orchestra.run.smithi185:> sudo hostname smithi185 2020-12-05T18:56:49.147 INFO:teuthology.orchestra.run.smithi185:> sudo sed -i -e 's/smithi185.front.sepia.ceph.com/smithi185/g' /etc/hosts 2020-12-05T18:56:49.163 INFO:teuthology.orchestra.run.smithi185:> sudo sed -i -e 's/172.21.15.185/172.21.15.185/g' /etc/hosts 2020-12-05T18:56:49.219 INFO:teuthology.orchestra.run.smithi185:> sudo sed -i -e 's/smithi185.front.sepia.ceph.com/smithi185/g' /etc/hostname 2020-12-05T18:56:49.274 INFO:teuthology.orchestra.run.smithi185:> sudo hostname smithi185 2020-12-05T18:56:49.328 INFO:teuthology.provision.fog.smithi185:Deploy complete! 2020-12-05T18:56:49.328 DEBUG:teuthology.parallel:result is None 2020-12-05T18:57:11.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:57:11.257 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-12-05T18:57:11.258 INFO:teuthology.orchestra.run.smithi023:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-12-05T18:57:11.655 INFO:teuthology.provision.fog.smithi023:Node is ready 2020-12-05T18:57:11.655 INFO:teuthology.orchestra.run.smithi023:> hostname 2020-12-05T18:57:11.702 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023.front.sepia.ceph.com 2020-12-05T18:57:11.702 INFO:teuthology.orchestra.run.smithi023:> grep smithi023.front.sepia.ceph.com /etc/hosts 2020-12-05T18:57:11.751 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23 smithi023.front.sepia.ceph.com smithi023 2020-12-05T18:57:11.751 INFO:teuthology.orchestra.run.smithi023:> sudo hostname smithi023 2020-12-05T18:57:11.889 INFO:teuthology.orchestra.run.smithi023:> sudo sed -i -e 's/smithi023.front.sepia.ceph.com/smithi023/g' /etc/hosts 2020-12-05T18:57:11.905 INFO:teuthology.orchestra.run.smithi023:> sudo sed -i -e 's/172.21.15.23/172.21.15.23/g' /etc/hosts 2020-12-05T18:57:11.964 INFO:teuthology.orchestra.run.smithi023:> sudo sed -i -e 's/smithi023.front.sepia.ceph.com/smithi023/g' /etc/hostname 2020-12-05T18:57:12.017 INFO:teuthology.orchestra.run.smithi023:> sudo hostname smithi023 2020-12-05T18:57:12.075 INFO:teuthology.provision.fog.smithi023:Deploy complete! 2020-12-05T18:57:12.075 DEBUG:teuthology.parallel:result is None 2020-12-05T18:57:12.221 INFO:teuthology.lock.ops:Checking smithi185.front.sepia.ceph.com 2020-12-05T18:57:12.222 INFO:teuthology.lock.ops:New key found. Updating... 2020-12-05T18:57:12.237 INFO:teuthology.lock.ops:Checking smithi023.front.sepia.ceph.com 2020-12-05T18:57:12.237 INFO:teuthology.lock.ops:New key found. Updating... 2020-12-05T18:57:12.250 INFO:teuthology.lock.ops:Updating [smithi185.front.sepia.ceph.com]: set os type and version on server 2020-12-05T18:57:12.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T18:57:12.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:57:12.326 INFO:teuthology.orchestra.run.smithi185:> true 2020-12-05T18:57:12.518 INFO:teuthology.orchestra.run.smithi185:> uname -m 2020-12-05T18:57:12.564 INFO:teuthology.orchestra.run.smithi185.stdout:x86_64 2020-12-05T18:57:12.565 INFO:teuthology.orchestra.run.smithi185:> cat /etc/os-release 2020-12-05T18:57:12.612 INFO:teuthology.orchestra.run.smithi185.stdout:NAME="Ubuntu" 2020-12-05T18:57:12.613 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-12-05T18:57:12.613 INFO:teuthology.orchestra.run.smithi185.stdout:ID=ubuntu 2020-12-05T18:57:12.613 INFO:teuthology.orchestra.run.smithi185.stdout:ID_LIKE=debian 2020-12-05T18:57:12.613 INFO:teuthology.orchestra.run.smithi185.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-12-05T18:57:12.614 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION_ID="18.04" 2020-12-05T18:57:12.614 INFO:teuthology.orchestra.run.smithi185.stdout:HOME_URL="https://www.ubuntu.com/" 2020-12-05T18:57:12.614 INFO:teuthology.orchestra.run.smithi185.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-12-05T18:57:12.614 INFO:teuthology.orchestra.run.smithi185.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-12-05T18:57:12.615 INFO:teuthology.orchestra.run.smithi185.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-12-05T18:57:12.615 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION_CODENAME=bionic 2020-12-05T18:57:12.615 INFO:teuthology.orchestra.run.smithi185.stdout:UBUNTU_CODENAME=bionic 2020-12-05T18:57:12.616 INFO:teuthology.lock.ops:Updating smithi185.front.sepia.ceph.com on lock server 2020-12-05T18:57:12.632 INFO:teuthology.lock.ops:Updating [smithi023.front.sepia.ceph.com]: set os type and version on server 2020-12-05T18:57:12.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T18:57:12.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:57:12.704 INFO:teuthology.orchestra.run.smithi023:> true 2020-12-05T18:57:12.903 INFO:teuthology.orchestra.run.smithi023:> uname -m 2020-12-05T18:57:12.950 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2020-12-05T18:57:12.950 INFO:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2020-12-05T18:57:12.998 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="Ubuntu" 2020-12-05T18:57:12.998 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-12-05T18:57:12.999 INFO:teuthology.orchestra.run.smithi023.stdout:ID=ubuntu 2020-12-05T18:57:12.999 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE=debian 2020-12-05T18:57:12.999 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-12-05T18:57:12.999 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="18.04" 2020-12-05T18:57:13.000 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.ubuntu.com/" 2020-12-05T18:57:13.000 INFO:teuthology.orchestra.run.smithi023.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-12-05T18:57:13.000 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-12-05T18:57:13.000 INFO:teuthology.orchestra.run.smithi023.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-12-05T18:57:13.000 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_CODENAME=bionic 2020-12-05T18:57:13.001 INFO:teuthology.orchestra.run.smithi023.stdout:UBUNTU_CODENAME=bionic 2020-12-05T18:57:13.001 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2020-12-05T18:57:13.016 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-12-05T18:57:13.051 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKicYR1RJs1KLLijJn5XjjMvdd2qAN7whGTrlf9Sts32c4jcSYLivxfaBDl1Z4PYs+admw/twsV80qfYKiRalrC985ss9a8o0NQ5vPQWTaufIwxUIsoUPF8QPg61GrF9pskx3PmsJP0hiN+7ycg290F6UtevZu5XE0vhaRjyshghE/6na6Fc2yGQV6ZGqHkNgLJUlUDqwf/w6JbaR6UoIuQW04UmolcsI6V6YXpri1/e1rd/HWgJ8wtUkRJ78uXHY/hwDz0cWiBtqWAk6wz/u83jFkF2wk7IwxA26ZWBkvoAbcLbhidRkKQUQ8kve6Guk9a19NFBknZ2Cmhv7IigsR smithi185.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjM27pravCWwd4wQHgsJ/bDwVrPOCjgkhWR5PVyCYF56CoY484Bm4Avg063INbNigMP3qdmtLolZjFoUoDcPvjP/qX4K89tPyG41YAU1u4sOKflbBZEhxGh/Rralqy0dEhpZgxDNaBSc+SdLOJOmebRsioCorcAza/21HBrVtNKGdVlbpHT00Kfb7LyKEpAlT4l9/I0zAj+hdW5B55R9tFsXGKThcmFcR2QD4hay4DKwqEvUAuwajvp+MlUd9ftcUxHsLmB+kLbl8xUKZ4l8iaTT9HVO0uBSUXUhAmjBpiQBtmO4clgigeaZNIfNhxILBWg2Ks98OJNi7BKVxHZXzH 2020-12-05T18:57:13.051 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-12-05T18:57:13.102 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-12-05T18:57:13.110 INFO:teuthology.task.internal:Saving configuration 2020-12-05T18:57:13.125 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-12-05T18:57:13.132 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-12-05T18:57:13.149 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-12-05 18:51:18.165039', 'description': '/home/teuthworker/archive/teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi/5667657', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ01KU2O21GJZ9j5qW7OrG4Y9wr86BO6pBtjfGDyrg6w', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi185.front.sepia.ceph.com'} 2020-12-05T18:57:13.165 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-12-05 18:51:18.165061', 'description': '/home/teuthworker/archive/teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi/5667657', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJmrxQ+6rbaYfF2GWyKg7eh/vKXLCBqwFeXC1B7K2pfx', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi023.front.sepia.ceph.com'} 2020-12-05T18:57:13.165 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-12-05T18:57:13.172 INFO:teuthology.task.internal:roles: ubuntu@smithi185.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-12-05T18:57:13.173 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-12-05T18:57:13.173 INFO:teuthology.run_tasks:Running task console_log... 2020-12-05T18:57:13.228 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd212a40840>, signals=[15]) 2020-12-05T18:57:13.229 INFO:teuthology.run_tasks:Running task internal.connect... 2020-12-05T18:57:13.237 INFO:teuthology.task.internal:Opening connections... 2020-12-05T18:57:13.237 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi185.front.sepia.ceph.com 2020-12-05T18:57:13.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:57:13.313 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T18:57:13.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:57:13.387 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-12-05T18:57:13.396 INFO:teuthology.orchestra.run.smithi185:> uname -m 2020-12-05T18:57:13.483 INFO:teuthology.orchestra.run.smithi185.stdout:x86_64 2020-12-05T18:57:13.483 INFO:teuthology.orchestra.run.smithi185:> cat /etc/os-release 2020-12-05T18:57:13.528 INFO:teuthology.orchestra.run.smithi185.stdout:NAME="Ubuntu" 2020-12-05T18:57:13.528 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-12-05T18:57:13.528 INFO:teuthology.orchestra.run.smithi185.stdout:ID=ubuntu 2020-12-05T18:57:13.528 INFO:teuthology.orchestra.run.smithi185.stdout:ID_LIKE=debian 2020-12-05T18:57:13.529 INFO:teuthology.orchestra.run.smithi185.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-12-05T18:57:13.529 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION_ID="18.04" 2020-12-05T18:57:13.529 INFO:teuthology.orchestra.run.smithi185.stdout:HOME_URL="https://www.ubuntu.com/" 2020-12-05T18:57:13.529 INFO:teuthology.orchestra.run.smithi185.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-12-05T18:57:13.529 INFO:teuthology.orchestra.run.smithi185.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-12-05T18:57:13.530 INFO:teuthology.orchestra.run.smithi185.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-12-05T18:57:13.530 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION_CODENAME=bionic 2020-12-05T18:57:13.530 INFO:teuthology.orchestra.run.smithi185.stdout:UBUNTU_CODENAME=bionic 2020-12-05T18:57:13.530 INFO:teuthology.lock.ops:Updating smithi185.front.sepia.ceph.com on lock server 2020-12-05T18:57:13.546 INFO:teuthology.orchestra.run.smithi023:> uname -m 2020-12-05T18:57:13.593 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2020-12-05T18:57:13.593 INFO:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2020-12-05T18:57:13.642 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="Ubuntu" 2020-12-05T18:57:13.642 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-12-05T18:57:13.642 INFO:teuthology.orchestra.run.smithi023.stdout:ID=ubuntu 2020-12-05T18:57:13.642 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE=debian 2020-12-05T18:57:13.643 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-12-05T18:57:13.643 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="18.04" 2020-12-05T18:57:13.643 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.ubuntu.com/" 2020-12-05T18:57:13.643 INFO:teuthology.orchestra.run.smithi023.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-12-05T18:57:13.643 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-12-05T18:57:13.644 INFO:teuthology.orchestra.run.smithi023.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-12-05T18:57:13.644 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_CODENAME=bionic 2020-12-05T18:57:13.644 INFO:teuthology.orchestra.run.smithi023.stdout:UBUNTU_CODENAME=bionic 2020-12-05T18:57:13.644 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2020-12-05T18:57:13.659 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-12-05T18:57:13.675 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-12-05T18:57:13.686 INFO:teuthology.task.internal:Checking for old test directory... 2020-12-05T18:57:13.686 INFO:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2020-12-05T18:57:13.688 INFO:teuthology.orchestra.run.smithi185:> test '!' -e /home/ubuntu/cephtest 2020-12-05T18:57:13.692 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-12-05T18:57:13.705 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-12-05T18:57:13.705 INFO:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2020-12-05T18:57:13.734 INFO:teuthology.orchestra.run.smithi185:> test -z $(ls -A /var/lib/ceph) 2020-12-05T18:57:13.765 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-12-05T18:57:13.810 INFO:teuthology.run_tasks:Running task kernel... 2020-12-05T18:57:13.828 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-12-05T18:57:13.828 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-12-05T18:57:13.829 INFO:teuthology.orchestra.run.smithi185:> uname -r 2020-12-05T18:57:13.835 INFO:teuthology.orchestra.run.smithi185.stdout:4.15.0-124-generic 2020-12-05T18:57:13.835 INFO:teuthology.task.kernel:Running kernel on smithi185: 4.15.0-124-generic 2020-12-05T18:57:13.835 INFO:teuthology.orchestra.run.smithi185:> sudo apt-get clean 2020-12-05T18:57:13.957 INFO:teuthology.orchestra.run.smithi185:> sudo apt-get update 2020-12-05T18:57:14.221 INFO:teuthology.orchestra.run.smithi185.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-12-05T18:57:14.221 INFO:teuthology.orchestra.run.smithi185.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-12-05T18:57:14.282 INFO:teuthology.orchestra.run.smithi185.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-12-05T18:57:14.535 INFO:teuthology.orchestra.run.smithi185.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-12-05T18:57:14.615 INFO:teuthology.orchestra.run.smithi185.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [870 kB] 2020-12-05T18:57:14.739 INFO:teuthology.orchestra.run.smithi185.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,789 kB] 2020-12-05T18:57:14.973 INFO:teuthology.orchestra.run.smithi185.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,453 kB] 2020-12-05T18:57:15.079 INFO:teuthology.orchestra.run.smithi185.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [284 kB] 2020-12-05T18:57:15.095 INFO:teuthology.orchestra.run.smithi185.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [184 kB] 2020-12-05T18:57:15.105 INFO:teuthology.orchestra.run.smithi185.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [24.4 kB] 2020-12-05T18:57:15.106 INFO:teuthology.orchestra.run.smithi185.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [968 kB] 2020-12-05T18:57:15.119 INFO:teuthology.orchestra.run.smithi185.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,172 kB] 2020-12-05T18:57:15.140 INFO:teuthology.orchestra.run.smithi185.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,092 kB] 2020-12-05T18:57:15.155 INFO:teuthology.orchestra.run.smithi185.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [376 kB] 2020-12-05T18:57:15.166 INFO:teuthology.orchestra.run.smithi185.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [206 kB] 2020-12-05T18:57:15.170 INFO:teuthology.orchestra.run.smithi185.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [244 kB] 2020-12-05T18:57:15.172 INFO:teuthology.orchestra.run.smithi185.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [28.0 kB] 2020-12-05T18:57:15.173 INFO:teuthology.orchestra.run.smithi185.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,548 kB] 2020-12-05T18:57:15.244 INFO:teuthology.orchestra.run.smithi185.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,695 kB] 2020-12-05T18:57:15.289 INFO:teuthology.orchestra.run.smithi185.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [357 kB] 2020-12-05T18:57:17.071 INFO:teuthology.orchestra.run.smithi185.stdout:Fetched 12.5 MB in 3s (4,336 kB/s) 2020-12-05T18:57:18.848 INFO:teuthology.orchestra.run.smithi185.stdout:Reading package lists... 2020-12-05T18:57:18.895 INFO:teuthology.orchestra.run.smithi185:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-12-05T18:57:18.990 INFO:teuthology.orchestra.run.smithi185.stdout:Reading package lists... 2020-12-05T18:57:19.220 INFO:teuthology.orchestra.run.smithi185.stdout:Building dependency tree... 2020-12-05T18:57:19.221 INFO:teuthology.orchestra.run.smithi185.stdout:Reading state information... 2020-12-05T18:57:19.300 INFO:teuthology.orchestra.run.smithi185.stderr:E: Unable to locate package linux-image-current-generic 2020-12-05T18:57:19.302 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-12-05T18:57:19.302 INFO:teuthology.orchestra.run.smithi185:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-12-05T18:57:19.387 INFO:teuthology.orchestra.run.smithi185.stdout:Reading package lists... 2020-12-05T18:57:19.639 INFO:teuthology.orchestra.run.smithi185.stdout:Building dependency tree... 2020-12-05T18:57:19.639 INFO:teuthology.orchestra.run.smithi185.stdout:Reading state information... 2020-12-05T18:57:19.814 INFO:teuthology.orchestra.run.smithi185.stdout:The following additional packages will be installed: 2020-12-05T18:57:19.814 INFO:teuthology.orchestra.run.smithi185.stdout: linux-generic linux-headers-4.15.0-126 linux-headers-4.15.0-126-generic 2020-12-05T18:57:19.815 INFO:teuthology.orchestra.run.smithi185.stdout: linux-headers-generic linux-image-4.15.0-126-generic 2020-12-05T18:57:19.815 INFO:teuthology.orchestra.run.smithi185.stdout: linux-modules-4.15.0-126-generic linux-modules-extra-4.15.0-126-generic 2020-12-05T18:57:19.816 INFO:teuthology.orchestra.run.smithi185.stdout:Suggested packages: 2020-12-05T18:57:19.817 INFO:teuthology.orchestra.run.smithi185.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-12-05T18:57:19.817 INFO:teuthology.orchestra.run.smithi185.stdout:Recommended packages: 2020-12-05T18:57:19.817 INFO:teuthology.orchestra.run.smithi185.stdout: thermald 2020-12-05T18:57:19.849 INFO:teuthology.orchestra.run.smithi185.stdout:The following NEW packages will be installed: 2020-12-05T18:57:19.850 INFO:teuthology.orchestra.run.smithi185.stdout: linux-headers-4.15.0-126 linux-headers-4.15.0-126-generic 2020-12-05T18:57:19.850 INFO:teuthology.orchestra.run.smithi185.stdout: linux-image-4.15.0-126-generic linux-modules-4.15.0-126-generic 2020-12-05T18:57:19.850 INFO:teuthology.orchestra.run.smithi185.stdout: linux-modules-extra-4.15.0-126-generic 2020-12-05T18:57:19.852 INFO:teuthology.orchestra.run.smithi185.stdout:The following packages will be upgraded: 2020-12-05T18:57:19.852 INFO:teuthology.orchestra.run.smithi185.stdout: linux-generic linux-headers-generic linux-image-generic 2020-12-05T18:57:20.026 INFO:teuthology.orchestra.run.smithi185.stdout:3 upgraded, 5 newly installed, 0 to remove and 7 not upgraded. 2020-12-05T18:57:20.026 INFO:teuthology.orchestra.run.smithi185.stdout:Need to get 65.9 MB of archives. 2020-12-05T18:57:20.026 INFO:teuthology.orchestra.run.smithi185.stdout:After this operation, 333 MB of additional disk space will be used. 2020-12-05T18:57:20.027 INFO:teuthology.orchestra.run.smithi185.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-126-generic amd64 4.15.0-126.129 [13.1 MB] 2020-12-05T18:57:21.063 INFO:teuthology.orchestra.run.smithi185.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-126-generic amd64 4.15.0-126.129 [8,013 kB] 2020-12-05T18:57:21.301 INFO:teuthology.orchestra.run.smithi185.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-126-generic amd64 4.15.0-126.129 [32.8 MB] 2020-12-05T18:57:22.249 INFO:teuthology.orchestra.run.smithi185.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.126.113 [1,872 B] 2020-12-05T18:57:22.249 INFO:teuthology.orchestra.run.smithi185.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.126.113 [2,620 B] 2020-12-05T18:57:22.249 INFO:teuthology.orchestra.run.smithi185.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-126 all 4.15.0-126.129 [10.9 MB] 2020-12-05T18:57:22.551 INFO:teuthology.orchestra.run.smithi185.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-126-generic amd64 4.15.0-126.129 [1,096 kB] 2020-12-05T18:57:22.590 INFO:teuthology.orchestra.run.smithi185.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.126.113 [2,524 B] 2020-12-05T18:57:23.107 INFO:teuthology.orchestra.run.smithi185.stdout:Fetched 65.9 MB in 3s (24.2 MB/s) 2020-12-05T18:57:23.280 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package linux-modules-4.15.0-126-generic. 2020-12-05T18:57:23.316 INFO:teuthology.orchestra.run.smithi185.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-12-05T18:57:23.319 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../0-linux-modules-4.15.0-126-generic_4.15.0-126.129_amd64.deb ... 2020-12-05T18:57:23.361 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking linux-modules-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:57:25.046 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package linux-image-4.15.0-126-generic. 2020-12-05T18:57:25.057 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../1-linux-image-4.15.0-126-generic_4.15.0-126.129_amd64.deb ... 2020-12-05T18:57:25.147 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking linux-image-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:57:25.649 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-126-generic. 2020-12-05T18:57:25.656 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-126-generic_4.15.0-126.129_amd64.deb ... 2020-12-05T18:57:25.699 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking linux-modules-extra-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:57:29.746 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../3-linux-generic_4.15.0.126.113_amd64.deb ... 2020-12-05T18:57:29.868 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking linux-generic (4.15.0.126.113) over (4.15.0.124.111) ... 2020-12-05T18:57:30.218 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.126.113_amd64.deb ... 2020-12-05T18:57:30.347 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking linux-image-generic (4.15.0.126.113) over (4.15.0.124.111) ... 2020-12-05T18:57:30.666 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package linux-headers-4.15.0-126. 2020-12-05T18:57:30.676 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../5-linux-headers-4.15.0-126_4.15.0-126.129_all.deb ... 2020-12-05T18:57:30.717 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking linux-headers-4.15.0-126 (4.15.0-126.129) ... 2020-12-05T18:57:35.588 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package linux-headers-4.15.0-126-generic. 2020-12-05T18:57:35.608 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../6-linux-headers-4.15.0-126-generic_4.15.0-126.129_amd64.deb ... 2020-12-05T18:57:35.646 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking linux-headers-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:57:38.009 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.126.113_amd64.deb ... 2020-12-05T18:57:38.131 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking linux-headers-generic (4.15.0.126.113) over (4.15.0.124.111) ... 2020-12-05T18:57:38.475 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up linux-headers-4.15.0-126 (4.15.0-126.129) ... 2020-12-05T18:57:38.628 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up linux-modules-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:57:39.171 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up linux-headers-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:57:39.297 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up linux-image-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:57:39.774 INFO:teuthology.orchestra.run.smithi185.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-126-generic 2020-12-05T18:57:39.774 INFO:teuthology.orchestra.run.smithi185.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-126-generic 2020-12-05T18:57:39.907 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up linux-headers-generic (4.15.0.126.113) ... 2020-12-05T18:57:40.033 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up linux-modules-extra-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:57:40.519 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up linux-image-generic (4.15.0.126.113) ... 2020-12-05T18:57:40.644 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up linux-generic (4.15.0.126.113) ... 2020-12-05T18:57:40.787 INFO:teuthology.orchestra.run.smithi185.stdout:Processing triggers for linux-image-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:57:40.948 INFO:teuthology.orchestra.run.smithi185.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-12-05T18:57:40.948 INFO:teuthology.orchestra.run.smithi185.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-126-generic 2020-12-05T18:57:56.811 INFO:teuthology.orchestra.run.smithi185.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-12-05T18:57:56.812 INFO:teuthology.orchestra.run.smithi185.stdout:Sourcing file `/etc/default/grub' 2020-12-05T18:57:56.852 INFO:teuthology.orchestra.run.smithi185.stdout:Generating grub configuration file ... 2020-12-05T18:57:57.150 INFO:teuthology.orchestra.run.smithi185.stdout:Found linux image: /boot/vmlinuz-4.15.0-126-generic 2020-12-05T18:57:57.198 INFO:teuthology.orchestra.run.smithi185.stdout:Found initrd image: /boot/initrd.img-4.15.0-126-generic 2020-12-05T18:57:57.514 INFO:teuthology.orchestra.run.smithi185.stdout:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-12-05T18:57:57.546 INFO:teuthology.orchestra.run.smithi185.stdout:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-12-05T18:57:58.425 INFO:teuthology.orchestra.run.smithi185.stdout:done 2020-12-05T18:57:58.841 INFO:teuthology.orchestra.run.smithi185:> dpkg -s linux-image-generic 2020-12-05T18:57:58.863 INFO:teuthology.orchestra.run.smithi185.stdout:Package: linux-image-generic 2020-12-05T18:57:58.863 INFO:teuthology.orchestra.run.smithi185.stdout:Status: install ok installed 2020-12-05T18:57:58.864 INFO:teuthology.orchestra.run.smithi185.stdout:Priority: optional 2020-12-05T18:57:58.864 INFO:teuthology.orchestra.run.smithi185.stdout:Section: kernel 2020-12-05T18:57:58.864 INFO:teuthology.orchestra.run.smithi185.stdout:Installed-Size: 16 2020-12-05T18:57:58.864 INFO:teuthology.orchestra.run.smithi185.stdout:Maintainer: Ubuntu Kernel Team 2020-12-05T18:57:58.865 INFO:teuthology.orchestra.run.smithi185.stdout:Architecture: amd64 2020-12-05T18:57:58.865 INFO:teuthology.orchestra.run.smithi185.stdout:Source: linux-meta 2020-12-05T18:57:58.865 INFO:teuthology.orchestra.run.smithi185.stdout:Version: 4.15.0.126.113 2020-12-05T18:57:58.865 INFO:teuthology.orchestra.run.smithi185.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-12-05T18:57:58.865 INFO:teuthology.orchestra.run.smithi185.stdout:Depends: linux-image-4.15.0-126-generic, linux-modules-extra-4.15.0-126-generic, linux-firmware, intel-microcode, amd64-microcode 2020-12-05T18:57:58.866 INFO:teuthology.orchestra.run.smithi185.stdout:Recommends: thermald 2020-12-05T18:57:58.866 INFO:teuthology.orchestra.run.smithi185.stdout:Description: Generic Linux kernel image 2020-12-05T18:57:58.866 INFO:teuthology.orchestra.run.smithi185.stdout: This package will always depend on the latest generic kernel image 2020-12-05T18:57:58.866 INFO:teuthology.orchestra.run.smithi185.stdout: available. 2020-12-05T18:57:58.867 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-124-generic Expected: 4.15.0-126-generic 2020-12-05T18:57:58.867 INFO:teuthology.orchestra.run.smithi023:> uname -r 2020-12-05T18:57:58.874 INFO:teuthology.orchestra.run.smithi023.stdout:4.15.0-124-generic 2020-12-05T18:57:58.874 INFO:teuthology.task.kernel:Running kernel on smithi023: 4.15.0-124-generic 2020-12-05T18:57:58.874 INFO:teuthology.orchestra.run.smithi023:> sudo apt-get clean 2020-12-05T18:57:58.996 INFO:teuthology.orchestra.run.smithi023:> sudo apt-get update 2020-12-05T18:57:59.266 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-12-05T18:57:59.266 INFO:teuthology.orchestra.run.smithi023.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-12-05T18:57:59.332 INFO:teuthology.orchestra.run.smithi023.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-12-05T18:57:59.636 INFO:teuthology.orchestra.run.smithi023.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-12-05T18:57:59.667 INFO:teuthology.orchestra.run.smithi023.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,453 kB] 2020-12-05T18:57:59.803 INFO:teuthology.orchestra.run.smithi023.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,172 kB] 2020-12-05T18:58:00.103 INFO:teuthology.orchestra.run.smithi023.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [870 kB] 2020-12-05T18:58:00.122 INFO:teuthology.orchestra.run.smithi023.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [284 kB] 2020-12-05T18:58:00.128 INFO:teuthology.orchestra.run.smithi023.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [184 kB] 2020-12-05T18:58:00.131 INFO:teuthology.orchestra.run.smithi023.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [24.4 kB] 2020-12-05T18:58:00.132 INFO:teuthology.orchestra.run.smithi023.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,092 kB] 2020-12-05T18:58:00.157 INFO:teuthology.orchestra.run.smithi023.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,789 kB] 2020-12-05T18:58:00.203 INFO:teuthology.orchestra.run.smithi023.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [968 kB] 2020-12-05T18:58:00.231 INFO:teuthology.orchestra.run.smithi023.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [244 kB] 2020-12-05T18:58:00.261 INFO:teuthology.orchestra.run.smithi023.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [376 kB] 2020-12-05T18:58:00.277 INFO:teuthology.orchestra.run.smithi023.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [206 kB] 2020-12-05T18:58:00.287 INFO:teuthology.orchestra.run.smithi023.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [28.0 kB] 2020-12-05T18:58:00.288 INFO:teuthology.orchestra.run.smithi023.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,695 kB] 2020-12-05T18:58:00.332 INFO:teuthology.orchestra.run.smithi023.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,548 kB] 2020-12-05T18:58:00.388 INFO:teuthology.orchestra.run.smithi023.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [357 kB] 2020-12-05T18:58:01.782 INFO:teuthology.orchestra.run.smithi023.stdout:Fetched 12.5 MB in 3s (4,885 kB/s) 2020-12-05T18:58:03.542 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2020-12-05T18:58:03.588 INFO:teuthology.orchestra.run.smithi023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-12-05T18:58:03.674 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2020-12-05T18:58:03.938 INFO:teuthology.orchestra.run.smithi023.stdout:Building dependency tree... 2020-12-05T18:58:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:Reading state information... 2020-12-05T18:58:04.029 INFO:teuthology.orchestra.run.smithi023.stderr:E: Unable to locate package linux-image-current-generic 2020-12-05T18:58:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-12-05T18:58:04.031 INFO:teuthology.orchestra.run.smithi023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-12-05T18:58:04.113 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2020-12-05T18:58:04.370 INFO:teuthology.orchestra.run.smithi023.stdout:Building dependency tree... 2020-12-05T18:58:04.371 INFO:teuthology.orchestra.run.smithi023.stdout:Reading state information... 2020-12-05T18:58:04.557 INFO:teuthology.orchestra.run.smithi023.stdout:The following additional packages will be installed: 2020-12-05T18:58:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: linux-generic linux-headers-4.15.0-126 linux-headers-4.15.0-126-generic 2020-12-05T18:58:04.557 INFO:teuthology.orchestra.run.smithi023.stdout: linux-headers-generic linux-image-4.15.0-126-generic 2020-12-05T18:58:04.558 INFO:teuthology.orchestra.run.smithi023.stdout: linux-modules-4.15.0-126-generic linux-modules-extra-4.15.0-126-generic 2020-12-05T18:58:04.559 INFO:teuthology.orchestra.run.smithi023.stdout:Suggested packages: 2020-12-05T18:58:04.559 INFO:teuthology.orchestra.run.smithi023.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-12-05T18:58:04.559 INFO:teuthology.orchestra.run.smithi023.stdout:Recommended packages: 2020-12-05T18:58:04.560 INFO:teuthology.orchestra.run.smithi023.stdout: thermald 2020-12-05T18:58:04.590 INFO:teuthology.orchestra.run.smithi023.stdout:The following NEW packages will be installed: 2020-12-05T18:58:04.590 INFO:teuthology.orchestra.run.smithi023.stdout: linux-headers-4.15.0-126 linux-headers-4.15.0-126-generic 2020-12-05T18:58:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: linux-image-4.15.0-126-generic linux-modules-4.15.0-126-generic 2020-12-05T18:58:04.591 INFO:teuthology.orchestra.run.smithi023.stdout: linux-modules-extra-4.15.0-126-generic 2020-12-05T18:58:04.592 INFO:teuthology.orchestra.run.smithi023.stdout:The following packages will be upgraded: 2020-12-05T18:58:04.593 INFO:teuthology.orchestra.run.smithi023.stdout: linux-generic linux-headers-generic linux-image-generic 2020-12-05T18:58:04.771 INFO:teuthology.orchestra.run.smithi023.stdout:3 upgraded, 5 newly installed, 0 to remove and 7 not upgraded. 2020-12-05T18:58:04.772 INFO:teuthology.orchestra.run.smithi023.stdout:Need to get 65.9 MB of archives. 2020-12-05T18:58:04.772 INFO:teuthology.orchestra.run.smithi023.stdout:After this operation, 333 MB of additional disk space will be used. 2020-12-05T18:58:04.772 INFO:teuthology.orchestra.run.smithi023.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-126-generic amd64 4.15.0-126.129 [13.1 MB] 2020-12-05T18:58:05.756 INFO:teuthology.orchestra.run.smithi023.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-126-generic amd64 4.15.0-126.129 [8,013 kB] 2020-12-05T18:58:05.996 INFO:teuthology.orchestra.run.smithi023.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-126-generic amd64 4.15.0-126.129 [32.8 MB] 2020-12-05T18:58:06.887 INFO:teuthology.orchestra.run.smithi023.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.126.113 [1,872 B] 2020-12-05T18:58:06.887 INFO:teuthology.orchestra.run.smithi023.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.126.113 [2,620 B] 2020-12-05T18:58:06.888 INFO:teuthology.orchestra.run.smithi023.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-126 all 4.15.0-126.129 [10.9 MB] 2020-12-05T18:58:07.190 INFO:teuthology.orchestra.run.smithi023.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-126-generic amd64 4.15.0-126.129 [1,096 kB] 2020-12-05T18:58:07.204 INFO:teuthology.orchestra.run.smithi023.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.126.113 [2,524 B] 2020-12-05T18:58:07.727 INFO:teuthology.orchestra.run.smithi023.stdout:Fetched 65.9 MB in 3s (25.4 MB/s) 2020-12-05T18:58:07.865 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-modules-4.15.0-126-generic. 2020-12-05T18:58:07.904 INFO:teuthology.orchestra.run.smithi023.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-12-05T18:58:07.907 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../0-linux-modules-4.15.0-126-generic_4.15.0-126.129_amd64.deb ... 2020-12-05T18:58:07.961 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-modules-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:58:09.797 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-image-4.15.0-126-generic. 2020-12-05T18:58:09.810 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../1-linux-image-4.15.0-126-generic_4.15.0-126.129_amd64.deb ... 2020-12-05T18:58:09.902 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-image-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:58:10.408 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-126-generic. 2020-12-05T18:58:10.420 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-126-generic_4.15.0-126.129_amd64.deb ... 2020-12-05T18:58:10.458 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-modules-extra-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:58:14.190 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../3-linux-generic_4.15.0.126.113_amd64.deb ... 2020-12-05T18:58:14.318 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-generic (4.15.0.126.113) over (4.15.0.124.111) ... 2020-12-05T18:58:14.643 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.126.113_amd64.deb ... 2020-12-05T18:58:14.763 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-image-generic (4.15.0.126.113) over (4.15.0.124.111) ... 2020-12-05T18:58:15.108 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-headers-4.15.0-126. 2020-12-05T18:58:15.120 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../5-linux-headers-4.15.0-126_4.15.0-126.129_all.deb ... 2020-12-05T18:58:15.158 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-headers-4.15.0-126 (4.15.0-126.129) ... 2020-12-05T18:58:19.863 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package linux-headers-4.15.0-126-generic. 2020-12-05T18:58:19.884 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../6-linux-headers-4.15.0-126-generic_4.15.0-126.129_amd64.deb ... 2020-12-05T18:58:19.921 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-headers-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:58:22.355 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.126.113_amd64.deb ... 2020-12-05T18:58:22.486 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking linux-headers-generic (4.15.0.126.113) over (4.15.0.124.111) ... 2020-12-05T18:58:22.804 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-headers-4.15.0-126 (4.15.0-126.129) ... 2020-12-05T18:58:22.940 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-modules-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:58:23.475 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-headers-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:58:23.601 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-image-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:58:24.052 INFO:teuthology.orchestra.run.smithi023.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-126-generic 2020-12-05T18:58:24.053 INFO:teuthology.orchestra.run.smithi023.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-126-generic 2020-12-05T18:58:24.186 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-headers-generic (4.15.0.126.113) ... 2020-12-05T18:58:24.296 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-modules-extra-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:58:24.764 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-image-generic (4.15.0.126.113) ... 2020-12-05T18:58:24.890 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up linux-generic (4.15.0.126.113) ... 2020-12-05T18:58:25.016 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for linux-image-4.15.0-126-generic (4.15.0-126.129) ... 2020-12-05T18:58:25.202 INFO:teuthology.orchestra.run.smithi023.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-12-05T18:58:25.203 INFO:teuthology.orchestra.run.smithi023.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-126-generic 2020-12-05T18:58:40.539 INFO:teuthology.orchestra.run.smithi023.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-12-05T18:58:40.539 INFO:teuthology.orchestra.run.smithi023.stdout:Sourcing file `/etc/default/grub' 2020-12-05T18:58:40.615 INFO:teuthology.orchestra.run.smithi023.stdout:Generating grub configuration file ... 2020-12-05T18:58:40.966 INFO:teuthology.orchestra.run.smithi023.stdout:Found linux image: /boot/vmlinuz-4.15.0-126-generic 2020-12-05T18:58:41.005 INFO:teuthology.orchestra.run.smithi023.stdout:Found initrd image: /boot/initrd.img-4.15.0-126-generic 2020-12-05T18:58:41.348 INFO:teuthology.orchestra.run.smithi023.stdout:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-12-05T18:58:41.376 INFO:teuthology.orchestra.run.smithi023.stdout:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-12-05T18:58:42.222 INFO:teuthology.orchestra.run.smithi023.stdout:done 2020-12-05T18:58:42.652 INFO:teuthology.orchestra.run.smithi023:> dpkg -s linux-image-generic 2020-12-05T18:58:42.717 INFO:teuthology.orchestra.run.smithi023.stdout:Package: linux-image-generic 2020-12-05T18:58:42.717 INFO:teuthology.orchestra.run.smithi023.stdout:Status: install ok installed 2020-12-05T18:58:42.718 INFO:teuthology.orchestra.run.smithi023.stdout:Priority: optional 2020-12-05T18:58:42.718 INFO:teuthology.orchestra.run.smithi023.stdout:Section: kernel 2020-12-05T18:58:42.718 INFO:teuthology.orchestra.run.smithi023.stdout:Installed-Size: 16 2020-12-05T18:58:42.718 INFO:teuthology.orchestra.run.smithi023.stdout:Maintainer: Ubuntu Kernel Team 2020-12-05T18:58:42.719 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: amd64 2020-12-05T18:58:42.719 INFO:teuthology.orchestra.run.smithi023.stdout:Source: linux-meta 2020-12-05T18:58:42.719 INFO:teuthology.orchestra.run.smithi023.stdout:Version: 4.15.0.126.113 2020-12-05T18:58:42.719 INFO:teuthology.orchestra.run.smithi023.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-12-05T18:58:42.719 INFO:teuthology.orchestra.run.smithi023.stdout:Depends: linux-image-4.15.0-126-generic, linux-modules-extra-4.15.0-126-generic, linux-firmware, intel-microcode, amd64-microcode 2020-12-05T18:58:42.720 INFO:teuthology.orchestra.run.smithi023.stdout:Recommends: thermald 2020-12-05T18:58:42.720 INFO:teuthology.orchestra.run.smithi023.stdout:Description: Generic Linux kernel image 2020-12-05T18:58:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: This package will always depend on the latest generic kernel image 2020-12-05T18:58:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: available. 2020-12-05T18:58:42.721 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-124-generic Expected: 4.15.0-126-generic 2020-12-05T18:58:42.721 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-12-05T18:58:42.721 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-12-05T18:58:42.721 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-12-05T18:58:42.722 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2020-12-05T18:58:42.722 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi185.front.sepia.ceph.com, path=None, version=distro) 2020-12-05T18:58:42.722 INFO:teuthology.orchestra.run.smithi185:> sudo apt-get clean 2020-12-05T18:58:42.814 INFO:teuthology.orchestra.run.smithi185:> sudo apt-get update 2020-12-05T18:58:43.060 INFO:teuthology.orchestra.run.smithi185.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-12-05T18:58:43.061 INFO:teuthology.orchestra.run.smithi185.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-12-05T18:58:43.131 INFO:teuthology.orchestra.run.smithi185.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-12-05T18:58:43.211 INFO:teuthology.orchestra.run.smithi185.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-12-05T18:58:44.916 INFO:teuthology.orchestra.run.smithi185.stdout:Reading package lists... 2020-12-05T18:58:44.963 INFO:teuthology.orchestra.run.smithi185:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-12-05T18:58:45.087 INFO:teuthology.orchestra.run.smithi185.stdout:Reading package lists... 2020-12-05T18:58:45.337 INFO:teuthology.orchestra.run.smithi185.stdout:Building dependency tree... 2020-12-05T18:58:45.338 INFO:teuthology.orchestra.run.smithi185.stdout:Reading state information... 2020-12-05T18:58:45.422 INFO:teuthology.orchestra.run.smithi185.stderr:E: Unable to locate package linux-image-current-generic 2020-12-05T18:58:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-12-05T18:58:45.424 INFO:teuthology.orchestra.run.smithi185:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-12-05T18:58:45.507 INFO:teuthology.orchestra.run.smithi185.stdout:Reading package lists... 2020-12-05T18:58:45.768 INFO:teuthology.orchestra.run.smithi185.stdout:Building dependency tree... 2020-12-05T18:58:45.768 INFO:teuthology.orchestra.run.smithi185.stdout:Reading state information... 2020-12-05T18:58:45.967 INFO:teuthology.orchestra.run.smithi185.stdout:linux-image-generic is already the newest version (4.15.0.126.113). 2020-12-05T18:58:45.967 INFO:teuthology.orchestra.run.smithi185.stdout:0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded. 2020-12-05T18:58:45.969 INFO:teuthology.orchestra.run.smithi185:> dpkg -s linux-image-generic 2020-12-05T18:58:45.992 INFO:teuthology.orchestra.run.smithi185.stdout:Package: linux-image-generic 2020-12-05T18:58:45.992 INFO:teuthology.orchestra.run.smithi185.stdout:Status: install ok installed 2020-12-05T18:58:45.992 INFO:teuthology.orchestra.run.smithi185.stdout:Priority: optional 2020-12-05T18:58:45.992 INFO:teuthology.orchestra.run.smithi185.stdout:Section: kernel 2020-12-05T18:58:45.993 INFO:teuthology.orchestra.run.smithi185.stdout:Installed-Size: 16 2020-12-05T18:58:45.993 INFO:teuthology.orchestra.run.smithi185.stdout:Maintainer: Ubuntu Kernel Team 2020-12-05T18:58:45.993 INFO:teuthology.orchestra.run.smithi185.stdout:Architecture: amd64 2020-12-05T18:58:45.993 INFO:teuthology.orchestra.run.smithi185.stdout:Source: linux-meta 2020-12-05T18:58:45.994 INFO:teuthology.orchestra.run.smithi185.stdout:Version: 4.15.0.126.113 2020-12-05T18:58:45.994 INFO:teuthology.orchestra.run.smithi185.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-12-05T18:58:45.994 INFO:teuthology.orchestra.run.smithi185.stdout:Depends: linux-image-4.15.0-126-generic, linux-modules-extra-4.15.0-126-generic, linux-firmware, intel-microcode, amd64-microcode 2020-12-05T18:58:45.994 INFO:teuthology.orchestra.run.smithi185.stdout:Recommends: thermald 2020-12-05T18:58:45.994 INFO:teuthology.orchestra.run.smithi185.stdout:Description: Generic Linux kernel image 2020-12-05T18:58:45.995 INFO:teuthology.orchestra.run.smithi185.stdout: This package will always depend on the latest generic kernel image 2020-12-05T18:58:45.995 INFO:teuthology.orchestra.run.smithi185.stdout: available. 2020-12-05T18:58:45.995 INFO:teuthology.orchestra.run.smithi185:> mktemp 2020-12-05T18:58:46.041 INFO:teuthology.orchestra.run.smithi185.stdout:/tmp/tmp.IVbY5COqTs 2020-12-05T18:58:46.041 INFO:teuthology.orchestra.run.smithi185:> sudo cp /boot/grub/grub.cfg /tmp/tmp.IVbY5COqTs 2020-12-05T18:58:46.098 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0666 /tmp/tmp.IVbY5COqTs 2020-12-05T18:58:46.218 DEBUG:teuthology.orchestra.remote:smithi185:/tmp/tmp.IVbY5COqTs is 8KB 2020-12-05T18:58:46.229 INFO:teuthology.orchestra.run.smithi185:> rm -fr /tmp/tmp.IVbY5COqTs 2020-12-05T18:58:46.235 INFO:teuthology.orchestra.run.smithi185:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-12-05T18:58:46.288 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T18:58:46.288 INFO:teuthology.orchestra.run.smithi185:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-12-05T18:58:46.289 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-12-05T18:58:46.336 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-12-05T18:58:46.347 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records in 2020-12-05T18:58:46.347 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records out 2020-12-05T18:58:46.347 INFO:teuthology.orchestra.run.smithi185.stderr:93 bytes copied, 3.4357e-05 s, 2.7 MB/s 2020-12-05T18:58:46.348 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-12-05T18:58:46.357 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-126-generic 2020-12-05T18:58:46.357 INFO:teuthology.orchestra.run.smithi185:> sudo update-grub 2020-12-05T18:58:47.688 INFO:teuthology.orchestra.run.smithi185.stderr:Sourcing file `/etc/default/grub' 2020-12-05T18:58:47.711 INFO:teuthology.orchestra.run.smithi185.stderr:Generating grub configuration file ... 2020-12-05T18:58:47.948 INFO:teuthology.orchestra.run.smithi185.stderr:Found linux image: /boot/vmlinuz-4.15.0-126-generic 2020-12-05T18:58:47.960 INFO:teuthology.orchestra.run.smithi185.stderr:Found initrd image: /boot/initrd.img-4.15.0-126-generic 2020-12-05T18:58:48.257 INFO:teuthology.orchestra.run.smithi185.stderr:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-12-05T18:58:48.270 INFO:teuthology.orchestra.run.smithi185.stderr:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-12-05T18:58:48.832 INFO:teuthology.orchestra.run.smithi185.stderr:done 2020-12-05T18:58:48.834 INFO:teuthology.orchestra.run.smithi185:> sudo shutdown -r now 2020-12-05T18:58:48.836 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2020-12-05T18:58:48.836 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=None, version=distro) 2020-12-05T18:58:48.837 INFO:teuthology.orchestra.run.smithi023:> sudo apt-get clean 2020-12-05T18:58:48.900 INFO:teuthology.orchestra.run.smithi023:> sudo apt-get update 2020-12-05T18:58:49.154 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-12-05T18:58:49.221 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-12-05T18:58:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-12-05T18:58:49.305 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-12-05T18:58:51.086 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2020-12-05T18:58:51.136 INFO:teuthology.orchestra.run.smithi023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-12-05T18:58:51.212 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2020-12-05T18:58:51.478 INFO:teuthology.orchestra.run.smithi023.stdout:Building dependency tree... 2020-12-05T18:58:51.479 INFO:teuthology.orchestra.run.smithi023.stdout:Reading state information... 2020-12-05T18:58:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:E: Unable to locate package linux-image-current-generic 2020-12-05T18:58:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-12-05T18:58:51.576 INFO:teuthology.orchestra.run.smithi023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-12-05T18:58:51.655 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2020-12-05T18:58:51.894 INFO:teuthology.orchestra.run.smithi023.stdout:Building dependency tree... 2020-12-05T18:58:51.895 INFO:teuthology.orchestra.run.smithi023.stdout:Reading state information... 2020-12-05T18:58:52.101 INFO:teuthology.orchestra.run.smithi023.stdout:linux-image-generic is already the newest version (4.15.0.126.113). 2020-12-05T18:58:52.102 INFO:teuthology.orchestra.run.smithi023.stdout:0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded. 2020-12-05T18:58:52.103 INFO:teuthology.orchestra.run.smithi023:> dpkg -s linux-image-generic 2020-12-05T18:58:52.125 INFO:teuthology.orchestra.run.smithi023.stdout:Package: linux-image-generic 2020-12-05T18:58:52.125 INFO:teuthology.orchestra.run.smithi023.stdout:Status: install ok installed 2020-12-05T18:58:52.126 INFO:teuthology.orchestra.run.smithi023.stdout:Priority: optional 2020-12-05T18:58:52.126 INFO:teuthology.orchestra.run.smithi023.stdout:Section: kernel 2020-12-05T18:58:52.126 INFO:teuthology.orchestra.run.smithi023.stdout:Installed-Size: 16 2020-12-05T18:58:52.126 INFO:teuthology.orchestra.run.smithi023.stdout:Maintainer: Ubuntu Kernel Team 2020-12-05T18:58:52.126 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: amd64 2020-12-05T18:58:52.127 INFO:teuthology.orchestra.run.smithi023.stdout:Source: linux-meta 2020-12-05T18:58:52.127 INFO:teuthology.orchestra.run.smithi023.stdout:Version: 4.15.0.126.113 2020-12-05T18:58:52.127 INFO:teuthology.orchestra.run.smithi023.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-12-05T18:58:52.127 INFO:teuthology.orchestra.run.smithi023.stdout:Depends: linux-image-4.15.0-126-generic, linux-modules-extra-4.15.0-126-generic, linux-firmware, intel-microcode, amd64-microcode 2020-12-05T18:58:52.127 INFO:teuthology.orchestra.run.smithi023.stdout:Recommends: thermald 2020-12-05T18:58:52.128 INFO:teuthology.orchestra.run.smithi023.stdout:Description: Generic Linux kernel image 2020-12-05T18:58:52.128 INFO:teuthology.orchestra.run.smithi023.stdout: This package will always depend on the latest generic kernel image 2020-12-05T18:58:52.128 INFO:teuthology.orchestra.run.smithi023.stdout: available. 2020-12-05T18:58:52.128 INFO:teuthology.orchestra.run.smithi023:> mktemp 2020-12-05T18:58:52.173 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.3UcWeZ6VYv 2020-12-05T18:58:52.173 INFO:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub/grub.cfg /tmp/tmp.3UcWeZ6VYv 2020-12-05T18:58:52.229 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.3UcWeZ6VYv 2020-12-05T18:58:52.351 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.3UcWeZ6VYv is 8KB 2020-12-05T18:58:52.362 INFO:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.3UcWeZ6VYv 2020-12-05T18:58:52.369 INFO:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-12-05T18:58:52.430 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T18:58:52.430 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-12-05T18:58:52.431 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-12-05T18:58:52.482 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-12-05T18:58:52.498 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-12-05T18:58:52.498 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-12-05T18:58:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:93 bytes copied, 3.4457e-05 s, 2.7 MB/s 2020-12-05T18:58:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-12-05T18:58:52.508 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-126-generic 2020-12-05T18:58:52.509 INFO:teuthology.orchestra.run.smithi023:> sudo update-grub 2020-12-05T18:58:53.868 INFO:teuthology.orchestra.run.smithi023.stderr:Sourcing file `/etc/default/grub' 2020-12-05T18:58:53.881 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2020-12-05T18:58:54.084 INFO:teuthology.orchestra.run.smithi023.stderr:Found linux image: /boot/vmlinuz-4.15.0-126-generic 2020-12-05T18:58:54.097 INFO:teuthology.orchestra.run.smithi023.stderr:Found initrd image: /boot/initrd.img-4.15.0-126-generic 2020-12-05T18:58:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-12-05T18:58:54.402 INFO:teuthology.orchestra.run.smithi023.stderr:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-12-05T18:58:54.977 INFO:teuthology.orchestra.run.smithi023.stderr:done 2020-12-05T18:58:54.978 INFO:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2020-12-05T18:59:25.010 INFO:teuthology.misc:Re-opening connections... 2020-12-05T18:59:25.011 INFO:teuthology.misc:trying to connect to ubuntu@smithi185.front.sepia.ceph.com 2020-12-05T18:59:25.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T18:59:25.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T18:59:43.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.185 2020-12-05T18:59:43.555 INFO:teuthology.misc:trying to connect to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T18:59:43.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T18:59:43.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:01.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2020-12-05T19:00:01.983 DEBUG:teuthology.misc:waited 36.97191786766052 2020-12-05T19:00:02.983 INFO:teuthology.misc:trying to connect to ubuntu@smithi185.front.sepia.ceph.com 2020-12-05T19:00:02.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:02.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:34.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.185 2020-12-05T19:00:34.499 INFO:teuthology.misc:trying to connect to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:00:34.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:34.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:37.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2020-12-05T19:00:37.570 DEBUG:teuthology.misc:waited 72.5598680973053 2020-12-05T19:00:38.572 INFO:teuthology.misc:trying to connect to ubuntu@smithi185.front.sepia.ceph.com 2020-12-05T19:00:38.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:38.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:38.791 INFO:teuthology.orchestra.run.smithi185:> true 2020-12-05T19:00:39.231 DEBUG:teuthology.misc:waited 74.22009181976318 2020-12-05T19:00:40.232 INFO:teuthology.misc:trying to connect to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:00:40.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:40.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:40.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2020-12-05T19:00:40.638 DEBUG:teuthology.misc:waited 75.62773966789246 2020-12-05T19:00:41.639 INFO:teuthology.misc:trying to connect to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:00:41.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:41.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:44.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2020-12-05T19:00:44.702 DEBUG:teuthology.misc:waited 79.69182825088501 2020-12-05T19:00:45.703 INFO:teuthology.misc:trying to connect to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:00:45.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:45.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:47.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2020-12-05T19:00:47.774 DEBUG:teuthology.misc:waited 82.7637870311737 2020-12-05T19:00:48.775 INFO:teuthology.misc:trying to connect to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:00:48.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:48.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:50.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2020-12-05T19:00:50.847 DEBUG:teuthology.misc:waited 85.83605241775513 2020-12-05T19:00:51.847 INFO:teuthology.misc:trying to connect to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:00:51.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:51.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:53.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2020-12-05T19:00:53.923 DEBUG:teuthology.misc:waited 88.91192436218262 2020-12-05T19:00:54.924 INFO:teuthology.misc:trying to connect to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:00:54.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:54.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:55.755 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2020-12-05T19:00:55.755 DEBUG:teuthology.misc:waited 90.74489259719849 2020-12-05T19:00:56.756 INFO:teuthology.misc:trying to connect to ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:00:56.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:00:56.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:00:57.032 INFO:teuthology.orchestra.run.smithi023:> true 2020-12-05T19:00:58.106 DEBUG:teuthology.misc:waited 93.0949285030365 2020-12-05T19:00:59.107 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2020-12-05T19:00:59.108 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-126-generic"... 2020-12-05T19:00:59.108 INFO:teuthology.orchestra.run.smithi185:> uname -r 2020-12-05T19:00:59.115 INFO:teuthology.orchestra.run.smithi185.stdout:4.15.0-126-generic 2020-12-05T19:00:59.116 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-126-generic vs 4.15.0-126-generic 2020-12-05T19:00:59.116 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-12-05T19:00:59.117 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-12-05T19:00:59.117 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2020-12-05T19:00:59.117 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-126-generic"... 2020-12-05T19:00:59.117 INFO:teuthology.orchestra.run.smithi023:> uname -r 2020-12-05T19:00:59.124 INFO:teuthology.orchestra.run.smithi023.stdout:4.15.0-126-generic 2020-12-05T19:00:59.124 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-126-generic vs 4.15.0-126-generic 2020-12-05T19:00:59.125 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-12-05T19:00:59.125 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-12-05T19:01:00.126 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-12-05T19:01:00.126 INFO:teuthology.orchestra.run.smithi185:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-12-05T19:01:00.255 INFO:teuthology.orchestra.run.smithi185.stdout:ttyS1 2020-12-05T19:01:00.274 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-12-05T19:01:00.275 INFO:teuthology.orchestra.run.smithi023:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-12-05T19:01:00.312 INFO:teuthology.orchestra.run.smithi023.stdout:ttyS1 2020-12-05T19:01:00.336 INFO:teuthology.run_tasks:Running task internal.base... 2020-12-05T19:01:00.345 INFO:teuthology.task.internal:Creating test directory... 2020-12-05T19:01:00.346 INFO:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-12-05T19:01:00.379 INFO:teuthology.orchestra.run.smithi185:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-12-05T19:01:00.386 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-12-05T19:01:00.396 INFO:teuthology.run_tasks:Running task internal.archive... 2020-12-05T19:01:00.409 INFO:teuthology.task.internal:Creating archive directory... 2020-12-05T19:01:00.409 INFO:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-12-05T19:01:00.427 INFO:teuthology.orchestra.run.smithi185:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-12-05T19:01:00.446 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-12-05T19:01:00.456 INFO:teuthology.task.internal:Enabling coredump saving... 2020-12-05T19:01:00.457 INFO:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-12-05T19:01:00.483 INFO:teuthology.orchestra.run.smithi185:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-12-05T19:01:00.503 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-12-05T19:01:00.510 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-12-05T19:01:00.511 INFO:teuthology.orchestra.run.smithi185.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-12-05T19:01:00.517 INFO:teuthology.orchestra.run.smithi185.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-12-05T19:01:00.518 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-12-05T19:01:00.527 INFO:teuthology.task.internal:Configuring sudo... 2020-12-05T19:01:00.527 INFO:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-12-05T19:01:00.555 INFO:teuthology.orchestra.run.smithi185:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-12-05T19:01:00.575 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-12-05T19:01:00.584 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-12-05T19:01:00.585 INFO:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-12-05T19:01:00.611 INFO:teuthology.orchestra.run.smithi185:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-12-05T19:01:00.621 INFO:teuthology.orchestra.run.smithi185:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-12-05T19:01:00.670 INFO:teuthology.orchestra.run.smithi185:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-12-05T19:01:00.718 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:01:00.719 INFO:teuthology.orchestra.run.smithi185:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-12-05T19:01:00.764 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-12-05T19:01:00.777 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records in 2020-12-05T19:01:00.777 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records out 2020-12-05T19:01:00.777 INFO:teuthology.orchestra.run.smithi185.stderr:152 bytes copied, 2.9829e-05 s, 5.1 MB/s 2020-12-05T19:01:00.778 INFO:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-12-05T19:01:00.785 INFO:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-12-05T19:01:00.831 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:01:00.832 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-12-05T19:01:00.877 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-12-05T19:01:00.895 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-12-05T19:01:00.896 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-12-05T19:01:00.896 INFO:teuthology.orchestra.run.smithi023.stderr:152 bytes copied, 3.1577e-05 s, 4.8 MB/s 2020-12-05T19:01:00.897 INFO:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2020-12-05T19:01:00.939 INFO:teuthology.orchestra.run.smithi185:> sudo service rsyslog restart 2020-12-05T19:01:00.985 INFO:teuthology.run_tasks:Running task internal.timer... 2020-12-05T19:01:00.998 INFO:teuthology.task.internal:Starting timer... 2020-12-05T19:01:00.999 INFO:teuthology.run_tasks:Running task pcp... 2020-12-05T19:01:01.015 INFO:teuthology.run_tasks:Running task selinux... 2020-12-05T19:01:01.032 DEBUG:teuthology.task.selinux:Excluding smithi185: OS 'ubuntu' does not support SELinux 2020-12-05T19:01:01.033 DEBUG:teuthology.task.selinux:Excluding smithi023: OS 'ubuntu' does not support SELinux 2020-12-05T19:01:01.033 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-12-05T19:01:01.034 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-12-05T19:01:01.034 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-12-05T19:01:01.035 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-12-05T19:01:01.045 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-12-05T19:01:01.046 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-12-05T19:01:01.059 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-12-05T19:01:01.060 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi185.front.sepia.ceph.com,smithi023.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-12-05T19:01:01.406 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-12-05T19:01:02.324 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-12-05T19:01:04.886 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-12-05T19:01:04.915 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:04 +0000 (0:00:00.078) 0:00:00.078 ***** 2020-12-05T19:01:04.932 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:04 +0000 (0:00:00.017) 0:00:00.095 ***** 2020-12-05T19:01:06.631 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-12-05T19:01:06.633 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] 2020-12-05T19:01:06.687 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] 2020-12-05T19:01:06.741 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:06 +0000 (0:00:01.808) 0:00:01.904 ***** 2020-12-05T19:01:06.816 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:06 +0000 (0:00:00.074) 0:00:01.978 ***** 2020-12-05T19:01:07.006 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-12-05T19:01:07.007 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-12-05T19:01:07.052 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:07 +0000 (0:00:00.236) 0:00:02.215 ***** 2020-12-05T19:01:07.088 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:07 +0000 (0:00:00.035) 0:00:02.251 ***** 2020-12-05T19:01:07.528 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-12-05T19:01:07.530 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-12-05T19:01:07.573 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:07 +0000 (0:00:00.485) 0:00:02.736 ***** 2020-12-05T19:01:07.581 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-12-05T19:01:07.622 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:07 +0000 (0:00:00.048) 0:00:02.785 ***** 2020-12-05T19:01:07.889 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-12-05T19:01:07.890 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.060485", "end": "2020-12-05 19:01:07.685511", "rc": 0, "start": "2020-12-05 19:01:07.625026", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.064427", "end": "2020-12-05 19:01:07.901110", "rc": 0, "start": "2020-12-05 19:01:07.836683", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-12-05T19:01:07.928 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:07 +0000 (0:00:00.306) 0:00:03.091 ***** 2020-12-05T19:01:07.960 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:07 +0000 (0:00:00.031) 0:00:03.123 ***** 2020-12-05T19:01:08.166 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-12-05T19:01:08.168 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-12-05T19:01:08.202 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:08 +0000 (0:00:00.241) 0:00:03.365 ***** 2020-12-05T19:01:08.234 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:08 +0000 (0:00:00.032) 0:00:03.397 ***** 2020-12-05T19:01:08.848 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-12-05T19:01:08.849 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1607194868.2484295-24418946899695/source", "state": "file", "uid": 0} changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1607194868.2161322-26664443663286/source", "state": "file", "uid": 0} 2020-12-05T19:01:08.882 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:08 +0000 (0:00:00.647) 0:00:04.045 ***** 2020-12-05T19:01:08.914 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:08 +0000 (0:00:00.031) 0:00:04.077 ***** 2020-12-05T19:01:09.188 INFO:teuthology.task.ansible.out: 2020-12-05T19:01:09.190 INFO:teuthology.task.ansible.out:TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCto 2020-12-05T19:01:09.191 INFO:teuthology.task.ansible.out:HDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cD 2020-12-05T19:01:09.193 INFO:teuthology.task.ansible.out:nCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2z 2020-12-05T19:01:09.194 INFO:teuthology.task.ansible.out:wUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-12-05T19:01:09.327 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-12-05T19:01:09.335 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-12-05T19:01:09.476 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:09 +0000 (0:00:00.562) 0:00:04.639 ***** 2020-12-05T19:01:09.508 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:09 +0000 (0:00:00.032) 0:00:04.671 ***** 2020-12-05T19:01:10.183 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-12-05T19:01:10.185 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] 2020-12-05T19:01:10.268 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-12-05T19:01:10.270 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:10 +0000 (0:00:00.759) 0:00:05.431 ***** 2020-12-05T19:01:10.279 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] 2020-12-05T19:01:10.360 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:10 +0000 (0:00:00.092) 0:00:05.523 ***** 2020-12-05T19:01:10.677 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-12-05T19:01:10.679 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.207746", "end": "2020-12-05 19:01:10.648196", "rc": 0, "start": "2020-12-05 19:01:10.440450", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-12-05T19:01:10.717 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:10 +0000 (0:00:00.356) 0:00:05.880 ***** 2020-12-05T19:01:10.734 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.166663", "end": "2020-12-05 19:01:10.709400", "rc": 0, "start": "2020-12-05 19:01:10.542737", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-12-05T19:01:10.770 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:10 +0000 (0:00:00.053) 0:00:05.933 ***** 2020-12-05T19:01:10.874 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-12-05T19:01:10.875 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013426", "end": "2020-12-05 19:01:10.874105", "failed_when_result": false, "rc": 0, "start": "2020-12-05 19:01:10.860679", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-12-05T19:01:10.918 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:10 +0000 (0:00:00.147) 0:00:06.081 ***** 2020-12-05T19:01:10.926 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014046", "end": "2020-12-05 19:01:10.716171", "failed_when_result": false, "rc": 0, "start": "2020-12-05 19:01:10.702125", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-12-05T19:01:10.964 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:10 +0000 (0:00:00.045) 0:00:06.126 ***** 2020-12-05T19:01:10.972 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-12-05T19:01:10.974 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-12-05T19:01:11.009 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.045) 0:00:06.172 ***** 2020-12-05T19:01:11.016 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-12-05T19:01:11.053 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.044) 0:00:06.216 ***** 2020-12-05T19:01:11.060 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-12-05T19:01:11.062 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.104 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.049) 0:00:06.266 ***** 2020-12-05T19:01:11.110 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.147 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.043) 0:00:06.310 ***** 2020-12-05T19:01:11.153 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.191 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.043) 0:00:06.354 ***** 2020-12-05T19:01:11.197 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.234 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.042) 0:00:06.397 ***** 2020-12-05T19:01:11.241 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-12-05T19:01:11.243 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.277 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.043) 0:00:06.440 ***** 2020-12-05T19:01:11.283 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.321 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.043) 0:00:06.483 ***** 2020-12-05T19:01:11.327 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.363 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.042) 0:00:06.526 ***** 2020-12-05T19:01:11.369 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.411 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.048) 0:00:06.574 ***** 2020-12-05T19:01:11.417 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-12-05T19:01:11.419 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.455 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.044) 0:00:06.618 ***** 2020-12-05T19:01:11.461 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.499 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:11 +0000 (0:00:00.044) 0:00:06.662 ***** 2020-12-05T19:01:11.510 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-12-05T19:01:11.512 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.524 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.525 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.536 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.538 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.548 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.550 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.562 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.563 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0 2020-12-05T19:01:11.565 INFO:teuthology.task.ansible.out:452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was Fal 2020-12-05T19:01:11.567 INFO:teuthology.task.ansible.out:se"} 2020-12-05T19:01:11.575 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.577 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ 2020-12-05T19:01:11.579 INFO:teuthology.task.ansible.out:+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.590 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.592 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZn 2020-12-05T19:01:11.594 INFO:teuthology.task.ansible.out:scdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.602 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.604 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@elect 2020-12-05T19:01:11.606 INFO:teuthology.task.ansible.out:ric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.616 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.618 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b02 2020-12-05T19:01:11.620 INFO:teuthology.task.ansible.out:62be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.628 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.630 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.641 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.646 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name 2020-12-05T19:01:11.648 INFO:teuthology.task.ansible.out:": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.654 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.656 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.666 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.669 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.678 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.680 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2020-12-05T19:01:11.682 INFO:teuthology.task.ansible.out:2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.693 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.695 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.705 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.707 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.711 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.712 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.713 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.713 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee 2020-12-05T19:01:11.715 INFO:teuthology.task.ansible.out:37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.725 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.726 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.736 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.738 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 2020-12-05T19:01:11.740 INFO:teuthology.task.ansible.out: 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.749 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.751 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.762 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.764 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.765 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.765 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.776 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.778 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.778 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", 2020-12-05T19:01:11.780 INFO:teuthology.task.ansible.out:"ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.791 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.793 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.793 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.794 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.806 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.807 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.819 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.820 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.821 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.822 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.824 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.834 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.836 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.847 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.849 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.849 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.860 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.862 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.872 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.873 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.873 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.875 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.885 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.887 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.888 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.898 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.900 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.910 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.912 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.922 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.924 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.924 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "p 2020-12-05T19:01:11.926 INFO:teuthology.task.ansible.out:nawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.936 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.937 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.948 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.950 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.951 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.962 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.963 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.966 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.977 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.979 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.989 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.990 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:11.991 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.002 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.004 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.004 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.015 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.016 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.017 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 2020-12-05T19:01:12.019 INFO:teuthology.task.ansible.out: 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.029 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.030 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.032 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.042 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.044 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.044 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.044 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ov 2020-12-05T19:01:12.046 INFO:teuthology.task.ansible.out:pn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.057 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.059 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.070 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.071 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.083 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.086 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.087 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.098 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.100 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.110 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.112 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.112 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.113 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.124 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.126 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.137 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.139 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.149 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.151 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.161 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.162 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.173 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.175 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.176 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": 2020-12-05T19:01:12.177 INFO:teuthology.task.ansible.out:"prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.187 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.189 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.200 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.202 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.214 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.215 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.216 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.218 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.229 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.231 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.233 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.245 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.247 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.258 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.260 INFO:teuthology.task.ansible.out: skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.262 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.273 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.275 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.285 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.286 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.304 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.306 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.342 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.842) 0:00:07.505 ***** 2020-12-05T19:01:12.349 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.368 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-12-05T19:01:12.405 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.063) 0:00:07.568 ***** 2020-12-05T19:01:12.440 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.034) 0:00:07.602 ***** 2020-12-05T19:01:12.445 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-12-05T19:01:12.447 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.486 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.046) 0:00:07.649 ***** 2020-12-05T19:01:12.493 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-12-05T19:01:12.534 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.047) 0:00:07.697 ***** 2020-12-05T19:01:12.540 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-12-05T19:01:12.542 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.601 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.066) 0:00:07.764 ***** 2020-12-05T19:01:12.608 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-12-05T19:01:12.609 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.664 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.063) 0:00:07.827 ***** 2020-12-05T19:01:12.671 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-12-05T19:01:12.673 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.720 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.055) 0:00:07.883 ***** 2020-12-05T19:01:12.727 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-12-05T19:01:12.728 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.763 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.043) 0:00:07.926 ***** 2020-12-05T19:01:12.770 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-12-05T19:01:12.772 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.808 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.044) 0:00:07.971 ***** 2020-12-05T19:01:12.815 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-12-05T19:01:12.816 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.852 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.043) 0:00:08.015 ***** 2020-12-05T19:01:12.858 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.896 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.043) 0:00:08.059 ***** 2020-12-05T19:01:12.902 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-12-05T19:01:12.903 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.941 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:12 +0000 (0:00:00.045) 0:00:08.104 ***** 2020-12-05T19:01:12.949 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi023.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.951 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.961 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.962 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.964 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:12.965 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.015 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:13 +0000 (0:00:00.073) 0:00:08.178 ***** 2020-12-05T19:01:13.023 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-12-05T19:01:13.025 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.025 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.026 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.026 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.027 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.027 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.027 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.028 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.028 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.039 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.041 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.051 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.053 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.064 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.065 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.077 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.078 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.089 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.100 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.102 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.102 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.104 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.104 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.106 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.106 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.106 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.107 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.118 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.119 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.131 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.132 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.142 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.144 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.156 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.158 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.162 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.164 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.205 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:13 +0000 (0:00:00.189) 0:00:08.368 ***** 2020-12-05T19:01:13.212 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.213 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.213 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.214 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.225 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.227 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.238 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.240 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.249 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.251 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.261 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.262 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.272 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.274 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.283 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.285 INFO:teuthology.task.ansible.out: 2020-12-05T19:01:13.286 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.288 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.298 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.299 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.300 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.310 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.312 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.321 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.323 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:13.364 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:13 +0000 (0:00:00.158) 0:00:08.526 ***** 2020-12-05T19:01:13.757 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-12-05T19:01:13.759 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-12-05T19:01:13.759 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-12-05T19:01:13.795 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:13 +0000 (0:00:00.431) 0:00:08.958 ***** 2020-12-05T19:01:13.829 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:13 +0000 (0:00:00.033) 0:00:08.992 ***** 2020-12-05T19:01:13.834 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-12-05T19:01:13.836 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => { "msg": "Host smithi023.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-12-05T19:01:13.875 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:13 +0000 (0:00:00.046) 0:00:09.038 ***** 2020-12-05T19:01:13.888 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => { "msg": "Host smithi185.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-12-05T19:01:13.923 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:13 +0000 (0:00:00.048) 0:00:09.086 ***** 2020-12-05T19:01:13.929 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-12-05T19:01:13.931 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-12-05T19:01:13.968 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:13 +0000 (0:00:00.044) 0:00:09.131 ***** 2020-12-05T19:01:13.974 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-12-05T19:01:14.010 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.042) 0:00:09.173 ***** 2020-12-05T19:01:14.016 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-12-05T19:01:14.017 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.053 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.042) 0:00:09.216 ***** 2020-12-05T19:01:14.059 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.094 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.041) 0:00:09.257 ***** 2020-12-05T19:01:14.100 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-12-05T19:01:14.102 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.138 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.043) 0:00:09.301 ***** 2020-12-05T19:01:14.144 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.189 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.050) 0:00:09.352 ***** 2020-12-05T19:01:14.195 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-12-05T19:01:14.197 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.240 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.051) 0:00:09.403 ***** 2020-12-05T19:01:14.248 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.287 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.046) 0:00:09.450 ***** 2020-12-05T19:01:14.295 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.330 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.043) 0:00:09.493 ***** 2020-12-05T19:01:14.336 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.397 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.067) 0:00:09.560 ***** 2020-12-05T19:01:14.403 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-12-05T19:01:14.404 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.440 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.042) 0:00:09.603 ***** 2020-12-05T19:01:14.446 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.482 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.042) 0:00:09.645 ***** 2020-12-05T19:01:14.488 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-12-05T19:01:14.490 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.529 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.046) 0:00:09.692 ***** 2020-12-05T19:01:14.535 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.570 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.041) 0:00:09.734 ***** 2020-12-05T19:01:14.577 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-12-05T19:01:14.579 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.613 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.042) 0:00:09.776 ***** 2020-12-05T19:01:14.619 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.655 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.042) 0:00:09.818 ***** 2020-12-05T19:01:14.660 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-12-05T19:01:14.662 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.698 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.042) 0:00:09.861 ***** 2020-12-05T19:01:14.704 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.741 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.043) 0:00:09.904 ***** 2020-12-05T19:01:14.746 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-12-05T19:01:14.747 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:14.748 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:14.791 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.049) 0:00:09.954 ***** 2020-12-05T19:01:14.799 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:14.800 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:14.843 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.052) 0:00:10.006 ***** 2020-12-05T19:01:14.851 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-12-05T19:01:14.853 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.888 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.045) 0:00:10.051 ***** 2020-12-05T19:01:14.896 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.935 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.046) 0:00:10.098 ***** 2020-12-05T19:01:14.941 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:14.979 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:14 +0000 (0:00:00.044) 0:00:10.142 ***** 2020-12-05T19:01:14.985 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.022 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.043) 0:00:10.185 ***** 2020-12-05T19:01:15.028 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-12-05T19:01:15.029 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:15.064 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.041) 0:00:10.227 ***** 2020-12-05T19:01:15.070 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:15.108 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.044) 0:00:10.271 ***** 2020-12-05T19:01:15.114 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-12-05T19:01:15.116 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.165 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.057) 0:00:10.328 ***** 2020-12-05T19:01:15.179 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.216 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.050) 0:00:10.379 ***** 2020-12-05T19:01:15.221 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-12-05T19:01:15.223 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:15.264 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.047) 0:00:10.427 ***** 2020-12-05T19:01:15.271 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:15.312 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.048) 0:00:10.475 ***** 2020-12-05T19:01:15.319 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-12-05T19:01:15.321 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.357 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.045) 0:00:10.520 ***** 2020-12-05T19:01:15.364 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.401 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.043) 0:00:10.563 ***** 2020-12-05T19:01:15.406 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-12-05T19:01:15.408 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.442 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.041) 0:00:10.605 ***** 2020-12-05T19:01:15.450 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.486 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.043) 0:00:10.649 ***** 2020-12-05T19:01:15.492 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-12-05T19:01:15.493 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.531 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.045) 0:00:10.694 ***** 2020-12-05T19:01:15.538 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.576 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.044) 0:00:10.739 ***** 2020-12-05T19:01:15.582 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-12-05T19:01:15.584 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.626 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.050) 0:00:10.789 ***** 2020-12-05T19:01:15.634 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.674 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.047) 0:00:10.837 ***** 2020-12-05T19:01:15.681 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-12-05T19:01:15.683 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:15.719 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.044) 0:00:10.882 ***** 2020-12-05T19:01:15.725 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:15.761 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.041) 0:00:10.924 ***** 2020-12-05T19:01:15.766 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-12-05T19:01:15.768 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.805 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.043) 0:00:10.968 ***** 2020-12-05T19:01:15.811 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.849 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.044) 0:00:11.012 ***** 2020-12-05T19:01:15.856 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.893 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.043) 0:00:11.056 ***** 2020-12-05T19:01:15.901 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.937 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.044) 0:00:11.100 ***** 2020-12-05T19:01:15.943 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-12-05T19:01:15.944 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:15.979 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:15 +0000 (0:00:00.042) 0:00:11.142 ***** 2020-12-05T19:01:15.985 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.021 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.041) 0:00:11.184 ***** 2020-12-05T19:01:16.026 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-12-05T19:01:16.028 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.063 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.042) 0:00:11.226 ***** 2020-12-05T19:01:16.069 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.106 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.042) 0:00:11.269 ***** 2020-12-05T19:01:16.113 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.149 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.042) 0:00:11.312 ***** 2020-12-05T19:01:16.156 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.192 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.043) 0:00:11.355 ***** 2020-12-05T19:01:16.199 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-12-05T19:01:16.200 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.238 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.045) 0:00:11.401 ***** 2020-12-05T19:01:16.245 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.281 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.043) 0:00:11.444 ***** 2020-12-05T19:01:16.287 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-12-05T19:01:16.288 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:16.324 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.042) 0:00:11.487 ***** 2020-12-05T19:01:16.330 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:16.369 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.045) 0:00:11.532 ***** 2020-12-05T19:01:16.376 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-12-05T19:01:16.378 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.439 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.069) 0:00:11.602 ***** 2020-12-05T19:01:16.447 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.483 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.043) 0:00:11.646 ***** 2020-12-05T19:01:16.488 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-12-05T19:01:16.490 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.528 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.045) 0:00:11.691 ***** 2020-12-05T19:01:16.534 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.570 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.041) 0:00:11.733 ***** 2020-12-05T19:01:16.577 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-12-05T19:01:16.578 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.613 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.042) 0:00:11.776 ***** 2020-12-05T19:01:16.620 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.657 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.043) 0:00:11.820 ***** 2020-12-05T19:01:16.662 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-12-05T19:01:16.664 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.674 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.712 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.055) 0:00:11.875 ***** 2020-12-05T19:01:16.719 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.721 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.760 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.047) 0:00:11.923 ***** 2020-12-05T19:01:16.768 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi023.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.806 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.046) 0:00:11.969 ***** 2020-12-05T19:01:16.812 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.814 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.849 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.043) 0:00:12.012 ***** 2020-12-05T19:01:16.855 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-12-05T19:01:16.857 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.905 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.056) 0:00:12.068 ***** 2020-12-05T19:01:16.916 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:16.954 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:16 +0000 (0:00:00.048) 0:00:12.117 ***** 2020-12-05T19:01:17.057 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-12-05T19:01:17.059 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.011570", "end": "2020-12-05 19:01:16.850632", "rc": 0, "start": "2020-12-05 19:01:16.839062", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-12-05T19:01:17.097 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.142) 0:00:12.260 ***** 2020-12-05T19:01:17.105 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.016117", "end": "2020-12-05 19:01:17.100503", "rc": 0, "start": "2020-12-05 19:01:17.084386", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-12-05T19:01:17.144 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.047) 0:00:12.307 ***** 2020-12-05T19:01:17.150 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-12-05T19:01:17.152 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.188 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.043) 0:00:12.350 ***** 2020-12-05T19:01:17.193 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.230 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.043) 0:00:12.393 ***** 2020-12-05T19:01:17.236 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-12-05T19:01:17.238 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.289 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.058) 0:00:12.452 ***** 2020-12-05T19:01:17.296 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.332 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.043) 0:00:12.495 ***** 2020-12-05T19:01:17.339 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-12-05T19:01:17.341 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.376 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.043) 0:00:12.539 ***** 2020-12-05T19:01:17.383 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.420 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.044) 0:00:12.583 ***** 2020-12-05T19:01:17.426 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-12-05T19:01:17.428 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.463 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.043) 0:00:12.626 ***** 2020-12-05T19:01:17.471 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.507 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.043) 0:00:12.670 ***** 2020-12-05T19:01:17.513 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-12-05T19:01:17.514 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.550 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.043) 0:00:12.713 ***** 2020-12-05T19:01:17.557 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.593 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.042) 0:00:12.756 ***** 2020-12-05T19:01:17.600 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-12-05T19:01:17.601 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.637 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.044) 0:00:12.800 ***** 2020-12-05T19:01:17.644 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.682 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.044) 0:00:12.845 ***** 2020-12-05T19:01:17.687 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-12-05T19:01:17.688 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.726 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.044) 0:00:12.889 ***** 2020-12-05T19:01:17.733 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:17.793 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:17 +0000 (0:00:00.067) 0:00:12.956 ***** 2020-12-05T19:01:18.549 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-12-05T19:01:18.551 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.642891", "end": "2020-12-05 19:01:18.548150", "rc": 0, "start": "2020-12-05 19:01:17.905259", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:01:18.599 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:18 +0000 (0:00:00.805) 0:00:13.762 ***** 2020-12-05T19:01:18.626 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-12-05T19:01:18.628 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:18.664 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:18 +0000 (0:00:00.064) 0:00:13.827 ***** 2020-12-05T19:01:18.714 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.850405", "end": "2020-12-05 19:01:18.509188", "rc": 0, "start": "2020-12-05 19:01:17.658783", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:01:18.755 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:18 +0000 (0:00:00.090) 0:00:13.918 ***** 2020-12-05T19:01:18.783 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-12-05T19:01:18.785 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:18.819 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:18 +0000 (0:00:00.064) 0:00:13.982 ***** 2020-12-05T19:01:18.825 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-12-05T19:01:18.827 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.040400", "end": "2020-12-05 19:01:18.819214", "rc": 0, "start": "2020-12-05 19:01:18.778814", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:01:18.863 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:18 +0000 (0:00:00.044) 0:00:14.026 ***** 2020-12-05T19:01:18.989 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.041694", "end": "2020-12-05 19:01:18.792669", "rc": 0, "start": "2020-12-05 19:01:18.750975", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:01:19.038 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:19 +0000 (0:00:00.174) 0:00:14.200 ***** 2020-12-05T19:01:22.901 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** ok: [smithi185.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:01:22.941 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:22 +0000 (0:00:03.903) 0:00:18.104 ***** 2020-12-05T19:01:23.093 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:01:23.131 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:23 +0000 (0:00:00.189) 0:00:18.294 ***** 2020-12-05T19:01:23.806 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-12-05T19:01:23.807 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:01:23.862 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:23 +0000 (0:00:00.731) 0:00:19.025 ***** 2020-12-05T19:01:23.892 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:23.930 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:23 +0000 (0:00:00.067) 0:00:19.092 ***** 2020-12-05T19:01:23.978 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:01:24.024 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:24 +0000 (0:00:00.091) 0:00:19.184 ***** 2020-12-05T19:01:24.048 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:24.086 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:24 +0000 (0:00:00.065) 0:00:19.249 ***** 2020-12-05T19:01:24.240 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-12-05T19:01:24.241 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-12-05T19:01:24.280 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:24 +0000 (0:00:00.193) 0:00:19.443 ***** 2020-12-05T19:01:24.392 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-12-05T19:01:24.431 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:24 +0000 (0:00:00.150) 0:00:19.594 ***** 2020-12-05T19:01:25.236 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-12-05T19:01:25.238 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:01:25.277 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:25 +0000 (0:00:00.845) 0:00:20.440 ***** 2020-12-05T19:01:25.304 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:01:25.345 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:25 +0000 (0:00:00.068) 0:00:20.508 ***** 2020-12-05T19:01:25.688 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-12-05T19:01:25.690 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-12-05T19:01:25.732 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-12-05T19:01:25.974 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-12-05T19:01:26.017 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-12-05T19:01:26.250 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-12-05T19:01:26.292 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:26 +0000 (0:00:00.946) 0:00:21.455 ***** 2020-12-05T19:01:26.300 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-12-05T19:01:26.343 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:26 +0000 (0:00:00.050) 0:00:21.506 ***** 2020-12-05T19:01:26.422 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-12-05T19:01:26.424 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-12-05T19:01:26.460 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:26 +0000 (0:00:00.116) 0:00:21.623 ***** 2020-12-05T19:01:26.475 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-12-05T19:01:26.515 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:26 +0000 (0:00:00.055) 0:00:21.678 ***** 2020-12-05T19:01:26.753 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-12-05T19:01:26.755 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2020-12-05T19:01:26.807 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2020-12-05T19:01:27.018 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-12-05T19:01:27.082 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-12-05T19:01:27.303 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-12-05T19:01:27.344 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.828) 0:00:22.506 ***** 2020-12-05T19:01:27.353 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-12-05T19:01:27.397 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.053) 0:00:22.560 ***** 2020-12-05T19:01:27.405 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-12-05T19:01:27.407 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-12-05T19:01:27.443 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.045) 0:00:22.606 ***** 2020-12-05T19:01:27.450 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-12-05T19:01:27.487 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.043) 0:00:22.650 ***** 2020-12-05T19:01:27.488 INFO:teuthology.task.ansible.out: 2020-12-05T19:01:27.494 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-12-05T19:01:27.496 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.553 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.066) 0:00:22.716 ***** 2020-12-05T19:01:27.560 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.601 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.048) 0:00:22.764 ***** 2020-12-05T19:01:27.608 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-12-05T19:01:27.610 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:27.645 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.043) 0:00:22.808 ***** 2020-12-05T19:01:27.651 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:27.689 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.044) 0:00:22.852 ***** 2020-12-05T19:01:27.696 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-12-05T19:01:27.698 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.709 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.747 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.058) 0:00:22.910 ***** 2020-12-05T19:01:27.754 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.755 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.766 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.769 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.809 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.061) 0:00:22.972 ***** 2020-12-05T19:01:27.817 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-12-05T19:01:27.819 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.871 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.061) 0:00:23.034 ***** 2020-12-05T19:01:27.880 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:27.922 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:27 +0000 (0:00:00.051) 0:00:23.085 ***** 2020-12-05T19:01:28.788 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-12-05T19:01:28.790 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:01:28.823 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:28 +0000 (0:00:00.901) 0:00:23.986 ***** 2020-12-05T19:01:28.831 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:01:28.868 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:28 +0000 (0:00:00.044) 0:00:24.031 ***** 2020-12-05T19:01:29.125 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-12-05T19:01:29.126 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-12-05T19:01:29.162 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:29 +0000 (0:00:00.293) 0:00:24.325 ***** 2020-12-05T19:01:29.179 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-12-05T19:01:29.232 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:29 +0000 (0:00:00.069) 0:00:24.395 ***** 2020-12-05T19:01:29.478 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-12-05T19:01:29.480 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-12-05T19:01:29.517 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:29 +0000 (0:00:00.285) 0:00:24.680 ***** 2020-12-05T19:01:29.534 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-12-05T19:01:29.573 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:29 +0000 (0:00:00.055) 0:00:24.736 ***** 2020-12-05T19:01:29.663 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-12-05T19:01:29.665 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-12-05T19:01:29.720 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-12-05T19:01:29.784 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-12-05T19:01:29.823 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:29 +0000 (0:00:00.250) 0:00:24.986 ***** 2020-12-05T19:01:29.830 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-12-05T19:01:29.873 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:29 +0000 (0:00:00.049) 0:00:25.035 ***** 2020-12-05T19:01:29.879 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-12-05T19:01:29.881 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:29.926 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:29 +0000 (0:00:00.053) 0:00:25.089 ***** 2020-12-05T19:01:29.934 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:29.983 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:29 +0000 (0:00:00.056) 0:00:25.146 ***** 2020-12-05T19:01:29.989 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-12-05T19:01:29.991 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:30.027 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:30 +0000 (0:00:00.043) 0:00:25.190 ***** 2020-12-05T19:01:30.032 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-12-05T19:01:30.070 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:30 +0000 (0:00:00.042) 0:00:25.233 ***** 2020-12-05T19:01:30.076 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-12-05T19:01:30.077 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:30.112 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:30 +0000 (0:00:00.042) 0:00:25.275 ***** 2020-12-05T19:01:30.118 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:30.157 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:30 +0000 (0:00:00.044) 0:00:25.320 ***** 2020-12-05T19:01:30.453 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-12-05T19:01:30.454 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-12-05T19:01:30.491 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:30 +0000 (0:00:00.334) 0:00:25.654 ***** 2020-12-05T19:01:30.498 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-12-05T19:01:30.534 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:30 +0000 (0:00:00.042) 0:00:25.697 ***** 2020-12-05T19:01:31.117 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-12-05T19:01:31.118 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2020-12-05 19:00:56 UTC", "ActiveEnterTimestampMonotonic": "20405502", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target -.mount sysinit.target var-run.mount network.target systemd-tmpfiles-setup.service systemd-journald.socket system.slice time-sync.target remote-fs.target basic.target local-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2020-12-05 19:00:56 UTC", "AssertTimestampMonotonic": "20404118", "Before": "getty@tty1.service xdm.service plymouth-quit.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2020-12-05 19:00:56 UTC", "ConditionTimestampMonotonic": "20404118", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-12-05T19:01:31.119 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1311", "ExecMainStartTimestamp": "Sat 2020-12-05 19:00:56 UTC", "ExecMainStartTimestampMonotonic": "20405467", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Sat 2020-12-05 19:00:56 UTC] ; stop_time=[n/a] ; pid=1311 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2020-12-05 19:00:56 UTC", "InactiveExitTimestampMonotonic": "20405502", "InvocationID": "669b10fd8f0c4a54b927f8625167619c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-12-05T19:01:31.121 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1311", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-12-05T19:01:31.122 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2020-12-05 19:00:56 UTC", "StateChangeTimestampMonotonic": "20405502", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2020-12-05 19:00:56 UTC", "WatchdogTimestampMonotonic": "20405501", "WatchdogUSec": "0"}} 2020-12-05T19:01:31.170 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.636) 0:00:26.333 ***** 2020-12-05T19:01:31.184 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2020-12-05 19:00:35 UTC", "ActiveEnterTimestampMonotonic": "18832662", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target systemd-journald.socket network.target nss-lookup.target basic.target var-run.mount local-fs.target systemd-tmpfiles-setup.service system.slice sysinit.target time-sync.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2020-12-05 19:00:35 UTC", "AssertTimestampMonotonic": "18831200", "Before": "shutdown.target multi-user.target plymouth-quit.service xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2020-12-05 19:00:35 UTC", "ConditionTimestampMonotonic": "18831200", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-12-05T19:01:31.185 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1350", "ExecMainStartTimestamp": "Sat 2020-12-05 19:00:35 UTC", "ExecMainStartTimestampMonotonic": "18832625", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Sat 2020-12-05 19:00:35 UTC] ; stop_time=[n/a] ; pid=1350 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2020-12-05 19:00:35 UTC", "InactiveExitTimestampMonotonic": "18832662", "InvocationID": "9e67f6fc23bf4d6b923cbc6e06c78940", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-12-05T19:01:31.187 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1350", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-12-05T19:01:31.188 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2020-12-05 19:00:35 UTC", "StateChangeTimestampMonotonic": "18832662", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2020-12-05 19:00:35 UTC", "WatchdogTimestampMonotonic": "18832661", "WatchdogUSec": "0"}} 2020-12-05T19:01:31.222 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.051) 0:00:26.385 ***** 2020-12-05T19:01:31.227 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-12-05T19:01:31.229 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.293 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.071) 0:00:26.456 ***** 2020-12-05T19:01:31.303 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.339 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.045) 0:00:26.502 ***** 2020-12-05T19:01:31.345 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-12-05T19:01:31.347 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.404 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.064) 0:00:26.567 ***** 2020-12-05T19:01:31.414 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.459 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.055) 0:00:26.622 ***** 2020-12-05T19:01:31.465 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-12-05T19:01:31.467 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.502 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.042) 0:00:26.665 ***** 2020-12-05T19:01:31.510 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.547 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.045) 0:00:26.710 ***** 2020-12-05T19:01:31.554 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-12-05T19:01:31.556 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.591 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.043) 0:00:26.754 ***** 2020-12-05T19:01:31.598 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.635 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.044) 0:00:26.798 ***** 2020-12-05T19:01:31.637 INFO:teuthology.task.ansible.out: 2020-12-05T19:01:31.642 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-12-05T19:01:31.644 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.681 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.045) 0:00:26.844 ***** 2020-12-05T19:01:31.687 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.726 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.045) 0:00:26.889 ***** 2020-12-05T19:01:31.732 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-12-05T19:01:31.734 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.770 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.043) 0:00:26.932 ***** 2020-12-05T19:01:31.776 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.813 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.043) 0:00:26.976 ***** 2020-12-05T19:01:31.818 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-12-05T19:01:31.820 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.855 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.042) 0:00:27.018 ***** 2020-12-05T19:01:31.862 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.898 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.042) 0:00:27.061 ***** 2020-12-05T19:01:31.905 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-12-05T19:01:31.906 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.940 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.042) 0:00:27.103 ***** 2020-12-05T19:01:31.946 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:31.986 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:31 +0000 (0:00:00.045) 0:00:27.149 ***** 2020-12-05T19:01:31.992 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-12-05T19:01:31.993 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:32.029 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.043) 0:00:27.192 ***** 2020-12-05T19:01:32.037 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:32.072 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.043) 0:00:27.235 ***** 2020-12-05T19:01:32.078 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-12-05T19:01:32.080 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:32.115 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.042) 0:00:27.278 ***** 2020-12-05T19:01:32.121 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:32.157 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.041) 0:00:27.320 ***** 2020-12-05T19:01:32.163 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-12-05T19:01:32.165 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-12-05T19:01:32.200 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.043) 0:00:27.363 ***** 2020-12-05T19:01:32.206 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-12-05T19:01:32.243 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.043) 0:00:27.406 ***** 2020-12-05T19:01:32.249 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-12-05T19:01:32.251 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-12-05T19:01:32.288 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.045) 0:00:27.451 ***** 2020-12-05T19:01:32.296 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-12-05T19:01:32.335 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.046) 0:00:27.498 ***** 2020-12-05T19:01:32.341 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-12-05T19:01:32.343 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-12-05T19:01:32.383 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.047) 0:00:27.546 ***** 2020-12-05T19:01:32.389 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-12-05T19:01:32.428 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.045) 0:00:27.591 ***** 2020-12-05T19:01:32.434 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-12-05T19:01:32.436 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-12-05T19:01:32.474 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.045) 0:00:27.637 ***** 2020-12-05T19:01:32.480 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-12-05T19:01:32.520 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.046) 0:00:27.683 ***** 2020-12-05T19:01:32.599 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-12-05T19:01:32.601 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-12-05T19:01:32.642 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.122) 0:00:27.805 ***** 2020-12-05T19:01:32.659 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-12-05T19:01:32.696 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.053) 0:00:27.859 ***** 2020-12-05T19:01:32.787 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-12-05T19:01:32.789 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "name": "ubuntu", "state": "present", "system": false} 2020-12-05T19:01:32.825 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.129) 0:00:27.988 ***** 2020-12-05T19:01:32.842 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "name": "ubuntu", "state": "present", "system": false} 2020-12-05T19:01:32.878 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.052) 0:00:28.041 ***** 2020-12-05T19:01:32.884 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-12-05T19:01:32.886 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:32.919 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.041) 0:00:28.082 ***** 2020-12-05T19:01:32.929 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:32.967 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:32 +0000 (0:00:00.047) 0:00:28.130 ***** 2020-12-05T19:01:32.974 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-12-05T19:01:32.975 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:33.014 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:33 +0000 (0:00:00.047) 0:00:28.177 ***** 2020-12-05T19:01:33.021 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:33.071 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:33 +0000 (0:00:00.057) 0:00:28.234 ***** 2020-12-05T19:01:33.165 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-12-05T19:01:33.167 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-12-05T19:01:33.204 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:33 +0000 (0:00:00.132) 0:00:28.367 ***** 2020-12-05T19:01:33.224 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-12-05T19:01:33.263 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:33 +0000 (0:00:00.058) 0:00:28.426 ***** 2020-12-05T19:01:33.689 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-12-05T19:01:33.691 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-12-05T19:01:33.727 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:33 +0000 (0:00:00.464) 0:00:28.890 ***** 2020-12-05T19:01:33.754 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-12-05T19:01:33.755 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi023"}, "changed": false} TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-12-05T19:01:33.755 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-12-05T19:01:33.791 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:33 +0000 (0:00:00.063) 0:00:28.954 ***** 2020-12-05T19:01:33.828 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:33 +0000 (0:00:00.037) 0:00:28.991 ***** 2020-12-05T19:01:33.856 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-12-05T19:01:33.857 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi185"}, "changed": false} 2020-12-05T19:01:33.892 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:33 +0000 (0:00:00.064) 0:00:29.055 ***** 2020-12-05T19:01:34.671 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-12-05T19:01:34.672 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi023.front.sepia.ceph.com", "ansible_hostname": "smithi023", "ansible_nodename": "smithi023"}, "changed": true, "name": "smithi023"} 2020-12-05T19:01:34.707 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:34 +0000 (0:00:00.814) 0:00:29.870 ***** 2020-12-05T19:01:34.819 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi185.front.sepia.ceph.com", "ansible_hostname": "smithi185", "ansible_nodename": "smithi185"}, "changed": true, "name": "smithi185"} 2020-12-05T19:01:34.870 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:34 +0000 (0:00:00.163) 0:00:30.033 ***** 2020-12-05T19:01:35.035 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-12-05T19:01:35.037 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-12-05T19:01:35.072 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:35 +0000 (0:00:00.201) 0:00:30.235 ***** 2020-12-05T19:01:35.193 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-12-05T19:01:35.231 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:35 +0000 (0:00:00.159) 0:00:30.394 ***** 2020-12-05T19:01:35.395 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-12-05T19:01:35.432 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:35 +0000 (0:00:00.200) 0:00:30.595 ***** 2020-12-05T19:01:35.544 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-12-05T19:01:35.583 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:35 +0000 (0:00:00.151) 0:00:30.746 ***** 2020-12-05T19:01:35.954 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-12-05T19:01:35.956 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-12-05T19:01:35.995 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:35 +0000 (0:00:00.411) 0:00:31.158 ***** 2020-12-05T19:01:36.022 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-12-05T19:01:36.024 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.058 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.062) 0:00:31.221 ***** 2020-12-05T19:01:36.084 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-12-05T19:01:36.086 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.120 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.062) 0:00:31.283 ***** 2020-12-05T19:01:36.127 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-12-05T19:01:36.129 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-12-05T19:01:36.163 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.042) 0:00:31.326 ***** 2020-12-05T19:01:36.170 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-12-05T19:01:36.172 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.205 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.042) 0:00:31.368 ***** 2020-12-05T19:01:36.212 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-12-05T19:01:36.214 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.251 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.045) 0:00:31.414 ***** 2020-12-05T19:01:36.257 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-12-05T19:01:36.259 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.294 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.042) 0:00:31.456 ***** 2020-12-05T19:01:36.300 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-12-05T19:01:36.302 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.339 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.043) 0:00:31.500 ***** 2020-12-05T19:01:36.343 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-12-05T19:01:36.344 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.379 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.042) 0:00:31.542 ***** 2020-12-05T19:01:36.389 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-12-05T19:01:36.390 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.425 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.045) 0:00:31.588 ***** 2020-12-05T19:01:36.432 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-12-05T19:01:36.434 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.467 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.042) 0:00:31.630 ***** 2020-12-05T19:01:36.473 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-12-05T19:01:36.475 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.509 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.042) 0:00:31.672 ***** 2020-12-05T19:01:36.514 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-12-05T19:01:36.516 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.551 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.040) 0:00:31.713 ***** 2020-12-05T19:01:36.555 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-12-05T19:01:36.557 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.592 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.041) 0:00:31.755 ***** 2020-12-05T19:01:36.598 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-12-05T19:01:36.600 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.639 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.047) 0:00:31.802 ***** 2020-12-05T19:01:36.645 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-12-05T19:01:36.647 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.681 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.041) 0:00:31.844 ***** 2020-12-05T19:01:36.796 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-12-05T19:01:36.830 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.148) 0:00:31.993 ***** 2020-12-05T19:01:36.836 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-12-05T19:01:36.837 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.872 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.042) 0:00:32.035 ***** 2020-12-05T19:01:36.879 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-12-05T19:01:36.915 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.042) 0:00:32.078 ***** 2020-12-05T19:01:36.921 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-12-05T19:01:36.923 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:36.957 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.042) 0:00:32.120 ***** 2020-12-05T19:01:36.963 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-12-05T19:01:36.998 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:36 +0000 (0:00:00.041) 0:00:32.161 ***** 2020-12-05T19:01:37.005 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-12-05T19:01:37.040 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.041) 0:00:32.203 ***** 2020-12-05T19:01:37.046 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-12-05T19:01:37.046 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.082 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.041) 0:00:32.245 ***** 2020-12-05T19:01:37.088 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-12-05T19:01:37.124 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:32.287 ***** 2020-12-05T19:01:37.129 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-12-05T19:01:37.131 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.166 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.041) 0:00:32.329 ***** 2020-12-05T19:01:37.172 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-12-05T19:01:37.208 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.041) 0:00:32.371 ***** 2020-12-05T19:01:37.215 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-12-05T19:01:37.216 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.250 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:32.413 ***** 2020-12-05T19:01:37.256 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-12-05T19:01:37.258 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.293 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:32.456 ***** 2020-12-05T19:01:37.298 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:01:37.299 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.337 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.044) 0:00:32.500 ***** 2020-12-05T19:01:37.343 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-12-05T19:01:37.345 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.384 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.046) 0:00:32.547 ***** 2020-12-05T19:01:37.390 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-12-05T19:01:37.392 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.429 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.044) 0:00:32.592 ***** 2020-12-05T19:01:37.435 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-12-05T19:01:37.437 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.475 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.045) 0:00:32.638 ***** 2020-12-05T19:01:37.481 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-12-05T19:01:37.483 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.521 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.046) 0:00:32.684 ***** 2020-12-05T19:01:37.528 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:01:37.529 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.567 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.045) 0:00:32.730 ***** 2020-12-05T19:01:37.572 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-12-05T19:01:37.574 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.609 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:32.772 ***** 2020-12-05T19:01:37.615 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.651 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:32.814 ***** 2020-12-05T19:01:37.658 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-12-05T19:01:37.660 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.694 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:32.857 ***** 2020-12-05T19:01:37.700 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-12-05T19:01:37.702 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.737 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:32.900 ***** 2020-12-05T19:01:37.742 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-12-05T19:01:37.744 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.779 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:32.942 ***** 2020-12-05T19:01:37.785 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-12-05T19:01:37.787 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.821 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:32.984 ***** 2020-12-05T19:01:37.829 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.864 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:33.027 ***** 2020-12-05T19:01:37.870 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-12-05T19:01:37.872 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.906 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.041) 0:00:33.069 ***** 2020-12-05T19:01:37.911 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-12-05T19:01:37.913 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.948 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:33.111 ***** 2020-12-05T19:01:37.955 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-12-05T19:01:37.957 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:37.991 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:37 +0000 (0:00:00.042) 0:00:33.154 ***** 2020-12-05T19:01:37.997 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-12-05T19:01:37.999 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.037 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.046) 0:00:33.200 ***** 2020-12-05T19:01:38.043 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-12-05T19:01:38.045 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.094 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.056) 0:00:33.257 ***** 2020-12-05T19:01:38.100 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-12-05T19:01:38.102 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.140 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.046) 0:00:33.303 ***** 2020-12-05T19:01:38.147 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-12-05T19:01:38.149 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.187 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.046) 0:00:33.350 ***** 2020-12-05T19:01:38.193 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-12-05T19:01:38.195 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.233 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.045) 0:00:33.396 ***** 2020-12-05T19:01:38.239 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-12-05T19:01:38.241 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.280 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.047) 0:00:33.443 ***** 2020-12-05T19:01:38.287 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-12-05T19:01:38.288 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.323 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.042) 0:00:33.486 ***** 2020-12-05T19:01:38.329 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-12-05T19:01:38.331 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.366 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.042) 0:00:33.529 ***** 2020-12-05T19:01:38.373 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-12-05T19:01:38.374 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.410 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.044) 0:00:33.573 ***** 2020-12-05T19:01:38.417 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-12-05T19:01:38.419 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.455 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.045) 0:00:33.618 ***** 2020-12-05T19:01:38.461 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-12-05T19:01:38.463 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.501 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.045) 0:00:33.664 ***** 2020-12-05T19:01:38.507 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-12-05T19:01:38.509 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.546 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.044) 0:00:33.709 ***** 2020-12-05T19:01:38.552 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-12-05T19:01:38.554 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.590 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.044) 0:00:33.753 ***** 2020-12-05T19:01:38.596 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-12-05T19:01:38.598 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.635 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.044) 0:00:33.798 ***** 2020-12-05T19:01:38.642 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-12-05T19:01:38.644 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.679 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.043) 0:00:33.842 ***** 2020-12-05T19:01:38.685 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-12-05T19:01:38.687 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.727 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.048) 0:00:33.890 ***** 2020-12-05T19:01:38.733 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-12-05T19:01:38.736 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.771 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.043) 0:00:33.934 ***** 2020-12-05T19:01:38.778 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-12-05T19:01:38.780 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.814 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.042) 0:00:33.977 ***** 2020-12-05T19:01:38.820 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-12-05T19:01:38.821 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.857 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.043) 0:00:34.020 ***** 2020-12-05T19:01:38.863 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-12-05T19:01:38.865 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.901 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.044) 0:00:34.064 ***** 2020-12-05T19:01:38.907 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-12-05T19:01:38.908 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.951 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.049) 0:00:34.113 ***** 2020-12-05T19:01:38.956 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-12-05T19:01:38.958 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:38.993 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:38 +0000 (0:00:00.042) 0:00:34.156 ***** 2020-12-05T19:01:39.020 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-12-05T19:01:39.021 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:39.056 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.062) 0:00:34.219 ***** 2020-12-05T19:01:39.082 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-12-05T19:01:39.084 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for custom repos] *************************************** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005215", "end": "2020-12-05 19:01:38.897962", "rc": 0, "start": "2020-12-05 19:01:38.892747", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:01:39.118 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.061) 0:00:34.281 ***** 2020-12-05T19:01:39.150 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.032) 0:00:34.313 ***** 2020-12-05T19:01:39.157 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-12-05T19:01:39.193 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.042) 0:00:34.356 ***** 2020-12-05T19:01:39.286 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-12-05T19:01:39.288 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005130", "end": "2020-12-05 19:01:39.269774", "rc": 0, "start": "2020-12-05 19:01:39.264644", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:01:39.324 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.131) 0:00:34.487 ***** 2020-12-05T19:01:39.352 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-12-05T19:01:39.388 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.064) 0:00:34.551 ***** 2020-12-05T19:01:39.522 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-12-05T19:01:39.524 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-12-05T19:01:39.564 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.176) 0:00:34.727 ***** 2020-12-05T19:01:39.592 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-12-05T19:01:39.594 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:39.639 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.074) 0:00:34.802 ***** 2020-12-05T19:01:39.837 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-12-05T19:01:39.841 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-12-05T19:01:39.875 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.236) 0:00:35.038 ***** 2020-12-05T19:01:39.901 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-12-05T19:01:39.904 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:39.938 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:39 +0000 (0:00:00.062) 0:00:35.101 ***** 2020-12-05T19:01:40.809 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-12-05T19:01:40.811 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-12-05T19:01:40.982 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-12-05T19:01:41.765 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-12-05T19:01:41.802 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:41 +0000 (0:00:01.863) 0:00:36.965 ***** 2020-12-05T19:01:41.860 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-12-05T19:01:41.915 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:41 +0000 (0:00:00.113) 0:00:37.078 ***** 2020-12-05T19:01:42.719 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-12-05T19:01:42.721 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:01:42.755 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:42 +0000 (0:00:00.840) 0:00:37.918 ***** 2020-12-05T19:01:42.783 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-12-05T19:01:42.819 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:42 +0000 (0:00:00.063) 0:00:37.982 ***** 2020-12-05T19:01:42.824 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-12-05T19:01:42.826 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:01:42.862 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:42 +0000 (0:00:00.042) 0:00:38.025 ***** 2020-12-05T19:01:42.889 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-12-05T19:01:42.951 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:42 +0000 (0:00:00.088) 0:00:38.114 ***** 2020-12-05T19:01:42.980 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-12-05T19:01:42.982 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.032445", "end": "2020-12-05 19:01:42.797738", "rc": 0, "start": "2020-12-05 19:01:42.765293", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:01:43.022 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:43 +0000 (0:00:00.070) 0:00:38.185 ***** 2020-12-05T19:01:43.125 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036167", "end": "2020-12-05 19:01:43.114058", "rc": 0, "start": "2020-12-05 19:01:43.077891", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:01:43.162 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:43 +0000 (0:00:00.140) 0:00:38.325 ***** 2020-12-05T19:01:45.959 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-12-05T19:01:45.960 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:01:45.995 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:45 +0000 (0:00:02.832) 0:00:41.158 ***** 2020-12-05T19:01:46.147 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:01:46.184 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:46 +0000 (0:00:00.189) 0:00:41.347 ***** 2020-12-05T19:01:54.515 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-12-05T19:01:54.517 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 3 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137608 files and directories 2020-12-05T19:01:54.518 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 3 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-12-05T19:01:54.520 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137608 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-12-05T19:01:54.573 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:54 +0000 (0:00:08.388) 0:00:49.736 ***** 2020-12-05T19:01:55.029 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 3 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137608 files and directories 2020-12-05T19:01:55.031 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 3 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-12-05T19:01:55.033 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137608 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-12-05T19:01:55.072 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:55 +0000 (0:00:00.497) 0:00:50.233 ***** 2020-12-05T19:01:55.517 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-12-05T19:01:55.518 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:01:55.567 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:55 +0000 (0:00:00.496) 0:00:50.730 ***** 2020-12-05T19:01:55.594 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-12-05T19:01:55.595 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:55.630 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:55 +0000 (0:00:00.063) 0:00:50.793 ***** 2020-12-05T19:01:55.980 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-12-05T19:01:55.982 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:01:56.016 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:56 +0000 (0:00:00.386) 0:00:51.179 ***** 2020-12-05T19:01:56.047 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-12-05T19:01:56.049 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:01:56.087 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:01:56 +0000 (0:00:00.070) 0:00:51.250 ***** 2020-12-05T19:02:03.894 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-12-05T19:02:03.895 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 3 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.34 [40.2 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.34 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.34 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.34 [871 kB]\nFetched 10.4 MB in 1s (9826 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-12-05T19:02:03.896 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137466 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-12-05T19:02:03.898 INFO:teuthology.task.ansible.out:1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 3 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.34 [40.2 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.34 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.34 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.34 [871 kB]", "Fetched 10.4 MB in 1s (9826 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-12-05T19:02:03.900 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137466 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-12-05T19:02:03.932 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:03 +0000 (0:00:07.844) 0:00:59.095 ***** 2020-12-05T19:02:04.864 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 3 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.34 [40.2 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.34 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.34 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.34 [871 kB]\nFetched 10.4 MB in 1s (9803 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-12-05T19:02:04.866 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137466 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-12-05T19:02:04.868 INFO:teuthology.task.ansible.out:1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 3 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.34 [40.2 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.34 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.34 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.34 [871 kB]", "Fetched 10.4 MB in 1s (9803 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-12-05T19:02:04.869 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137466 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.34_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:02:04.901 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:04 +0000 (0:00:00.968) 0:01:00.064 ***** 2020-12-05T19:02:04.935 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:04 +0000 (0:00:00.033) 0:01:00.098 ***** 2020-12-05T19:02:05.055 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-12-05T19:02:05.057 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-12-05T19:02:05.093 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:05 +0000 (0:00:00.158) 0:01:00.256 ***** 2020-12-05T19:02:05.120 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-12-05T19:02:05.170 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:05.172 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:05 +0000 (0:00:00.079) 0:01:00.335 ***** 2020-12-05T19:02:05.319 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-12-05T19:02:05.321 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-12-05T19:02:05.380 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:05 +0000 (0:00:00.207) 0:01:00.543 ***** 2020-12-05T19:02:05.689 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-12-05T19:02:05.691 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-12-05T19:02:05.727 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:05 +0000 (0:00:00.347) 0:01:00.890 ***** 2020-12-05T19:02:05.808 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:02:05.864 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:05 +0000 (0:00:00.137) 0:01:01.027 ***** 2020-12-05T19:02:05.891 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-12-05T19:02:05.892 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1235, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-12-05T19:02:05.932 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:05 +0000 (0:00:00.068) 0:01:01.095 ***** 2020-12-05T19:02:05.991 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-12-05T19:02:05.993 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-12-05T19:02:06.028 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.095) 0:01:01.191 ***** 2020-12-05T19:02:06.055 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi185.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-12-05T19:02:06.057 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:06.094 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.066) 0:01:01.257 ***** 2020-12-05T19:02:06.129 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.034) 0:01:01.292 ***** 2020-12-05T19:02:06.218 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-12-05T19:02:06.220 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-12-05T19:02:06.259 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.129) 0:01:01.422 ***** 2020-12-05T19:02:06.421 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-12-05T19:02:06.422 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sat 2020-12-05 19:00:35 UTC", "ActiveEnterTimestampMonotonic": "19156773", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice remote-fs.target nss-lookup.target -.mount network.target systemd-tmpfiles-setup.service systemd-journald.socket basic.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2020-12-05 19:00:35 UTC", "AssertTimestampMonotonic": "18830124", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2020-12-05 19:00:35 UTC", "ConditionTimestampMonotonic": "18830124", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-12-05T19:02:06.424 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1406", "ExecMainStartTimestamp": "Sat 2020-12-05 19:00:35 UTC", "ExecMainStartTimestampMonotonic": "19156762", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2020-12-05 19:00:35 UTC", "InactiveExitTimestampMonotonic": "18831138", "InvocationID": "c4c12f79ae3a403b93e96b05dd92dd44", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-12-05T19:02:06.426 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1406", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-12-05T19:02:06.428 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2020-12-05 19:00:35 UTC", "StateChangeTimestampMonotonic": "19156773", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2020-12-05 19:00:35 UTC", "WatchdogTimestampMonotonic": "19156772", "WatchdogUSec": "0"}} 2020-12-05T19:02:06.460 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.200) 0:01:01.623 ***** 2020-12-05T19:02:06.486 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-12-05T19:02:06.488 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:06.531 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.071) 0:01:01.694 ***** 2020-12-05T19:02:06.558 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-12-05T19:02:06.560 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-12-05T19:02:06.593 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.062) 0:01:01.756 ***** 2020-12-05T19:02:06.628 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.034) 0:01:01.791 ***** 2020-12-05T19:02:06.654 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-12-05T19:02:06.656 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:06.692 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.063) 0:01:01.855 ***** 2020-12-05T19:02:06.719 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-12-05T19:02:06.720 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:06.754 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.061) 0:01:01.917 ***** 2020-12-05T19:02:06.770 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-12-05T19:02:06.772 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1235, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-12-05T19:02:06.806 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.051) 0:01:01.969 ***** 2020-12-05T19:02:06.812 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-12-05T19:02:06.814 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:06.852 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.045) 0:01:02.015 ***** 2020-12-05T19:02:06.878 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-12-05T19:02:06.880 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:06.913 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.061) 0:01:02.076 ***** 2020-12-05T19:02:06.942 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-12-05T19:02:06.944 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-12-05T19:02:06.978 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:06 +0000 (0:00:00.064) 0:01:02.141 ***** 2020-12-05T19:02:07.011 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.032) 0:01:02.174 ***** 2020-12-05T19:02:07.037 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-12-05T19:02:07.039 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.073 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.062) 0:01:02.236 ***** 2020-12-05T19:02:07.100 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:07.101 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.134 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.060) 0:01:02.297 ***** 2020-12-05T19:02:07.160 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-12-05T19:02:07.162 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.196 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.061) 0:01:02.359 ***** 2020-12-05T19:02:07.222 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-12-05T19:02:07.224 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.261 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.065) 0:01:02.424 ***** 2020-12-05T19:02:07.271 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-12-05T19:02:07.272 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sat 2020-12-05 19:00:56 UTC", "ActiveEnterTimestampMonotonic": "20705720", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target network.target basic.target system.slice nss-lookup.target sysinit.target -.mount systemd-journald.socket systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2020-12-05 19:00:56 UTC", "AssertTimestampMonotonic": "20401291", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2020-12-05 19:00:56 UTC", "ConditionTimestampMonotonic": "20401291", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-12-05T19:02:07.274 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1366", "ExecMainStartTimestamp": "Sat 2020-12-05 19:00:56 UTC", "ExecMainStartTimestampMonotonic": "20705709", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2020-12-05 19:00:56 UTC", "InactiveExitTimestampMonotonic": "20401949", "InvocationID": "b95509cc557647b59e68364f69573976", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-12-05T19:02:07.275 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1366", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-12-05T19:02:07.277 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2020-12-05 19:00:56 UTC", "StateChangeTimestampMonotonic": "20705720", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2020-12-05 19:00:56 UTC", "WatchdogTimestampMonotonic": "20705719", "WatchdogUSec": "0"}} 2020-12-05T19:02:07.311 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.049) 0:01:02.474 ***** 2020-12-05T19:02:07.317 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-12-05T19:02:07.319 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.352 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.041) 0:01:02.515 ***** 2020-12-05T19:02:07.358 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-12-05T19:02:07.360 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.394 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.041) 0:01:02.557 ***** 2020-12-05T19:02:07.400 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-12-05T19:02:07.402 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.436 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.041) 0:01:02.599 ***** 2020-12-05T19:02:07.442 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-12-05T19:02:07.444 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.477 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.040) 0:01:02.640 ***** 2020-12-05T19:02:07.483 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-12-05T19:02:07.485 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.522 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.044) 0:01:02.685 ***** 2020-12-05T19:02:07.527 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-12-05T19:02:07.529 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.564 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.041) 0:01:02.727 ***** 2020-12-05T19:02:07.570 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-12-05T19:02:07.571 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.607 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.043) 0:01:02.770 ***** 2020-12-05T19:02:07.614 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-12-05T19:02:07.615 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.650 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.043) 0:01:02.813 ***** 2020-12-05T19:02:07.657 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-12-05T19:02:07.658 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.694 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.043) 0:01:02.857 ***** 2020-12-05T19:02:07.700 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.738 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.044) 0:01:02.901 ***** 2020-12-05T19:02:07.745 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-12-05T19:02:07.747 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.781 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.042) 0:01:02.944 ***** 2020-12-05T19:02:07.787 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-12-05T19:02:07.789 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.823 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.042) 0:01:02.986 ***** 2020-12-05T19:02:07.830 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-12-05T19:02:07.832 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.886 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.062) 0:01:03.049 ***** 2020-12-05T19:02:07.892 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-12-05T19:02:07.894 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.931 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.045) 0:01:03.094 ***** 2020-12-05T19:02:07.938 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:07.978 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:07 +0000 (0:00:00.046) 0:01:03.141 ***** 2020-12-05T19:02:07.984 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-12-05T19:02:07.985 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.021 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.042) 0:01:03.184 ***** 2020-12-05T19:02:08.027 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-12-05T19:02:08.028 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.063 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.042) 0:01:03.226 ***** 2020-12-05T19:02:08.069 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:08.070 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.106 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.042) 0:01:03.269 ***** 2020-12-05T19:02:08.114 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-12-05T19:02:08.116 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.151 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.044) 0:01:03.314 ***** 2020-12-05T19:02:08.157 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-12-05T19:02:08.158 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.198 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.047) 0:01:03.361 ***** 2020-12-05T19:02:08.204 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-12-05T19:02:08.206 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.239 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.041) 0:01:03.402 ***** 2020-12-05T19:02:08.245 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-12-05T19:02:08.246 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.281 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.041) 0:01:03.444 ***** 2020-12-05T19:02:08.287 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-12-05T19:02:08.289 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.323 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.042) 0:01:03.486 ***** 2020-12-05T19:02:08.329 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-12-05T19:02:08.331 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.364 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.041) 0:01:03.527 ***** 2020-12-05T19:02:08.370 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-12-05T19:02:08.372 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.407 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.042) 0:01:03.570 ***** 2020-12-05T19:02:08.412 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-12-05T19:02:08.414 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.446 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.039) 0:01:03.609 ***** 2020-12-05T19:02:08.453 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-12-05T19:02:08.454 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.486 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.040) 0:01:03.649 ***** 2020-12-05T19:02:08.492 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-12-05T19:02:08.493 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.526 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.039) 0:01:03.689 ***** 2020-12-05T19:02:08.531 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-12-05T19:02:08.533 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.570 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.044) 0:01:03.733 ***** 2020-12-05T19:02:08.577 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-12-05T19:02:08.579 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.615 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.044) 0:01:03.778 ***** 2020-12-05T19:02:08.646 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-12-05T19:02:08.648 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.681 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.066) 0:01:03.844 ***** 2020-12-05T19:02:08.698 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-12-05T19:02:08.699 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-12-05T19:02:08.731 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.049) 0:01:03.894 ***** 2020-12-05T19:02:08.737 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-12-05T19:02:08.738 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.772 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.040) 0:01:03.935 ***** 2020-12-05T19:02:08.798 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-12-05T19:02:08.800 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.831 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.059) 0:01:03.994 ***** 2020-12-05T19:02:08.863 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-12-05T19:02:08.864 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.895 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.064) 0:01:04.058 ***** 2020-12-05T19:02:08.932 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-12-05T19:02:08.934 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:08.965 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:08 +0000 (0:00:00.069) 0:01:04.128 ***** 2020-12-05T19:02:08.992 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-12-05T19:02:08.994 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:09.025 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.059) 0:01:04.188 ***** 2020-12-05T19:02:09.031 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-12-05T19:02:09.033 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-12-05T19:02:09.066 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.041) 0:01:04.229 ***** 2020-12-05T19:02:09.072 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-12-05T19:02:09.073 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:09.106 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.039) 0:01:04.269 ***** 2020-12-05T19:02:09.111 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-12-05T19:02:09.113 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:09.145 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.039) 0:01:04.308 ***** 2020-12-05T19:02:09.150 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-12-05T19:02:09.152 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:09.185 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.039) 0:01:04.348 ***** 2020-12-05T19:02:09.211 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-12-05T19:02:09.213 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:09.243 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.058) 0:01:04.406 ***** 2020-12-05T19:02:09.269 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-12-05T19:02:09.271 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:09.301 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.057) 0:01:04.464 ***** 2020-12-05T19:02:09.326 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-12-05T19:02:09.328 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:09.359 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.057) 0:01:04.522 ***** 2020-12-05T19:02:09.477 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-12-05T19:02:09.479 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-12-05T19:02:09.513 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.154) 0:01:04.676 ***** 2020-12-05T19:02:09.552 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.038) 0:01:04.715 ***** 2020-12-05T19:02:09.829 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-12-05T19:02:09.831 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-12-05T19:02:09.866 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.313) 0:01:05.029 ***** 2020-12-05T19:02:09.872 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-12-05T19:02:09.873 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-12-05T19:02:09.909 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.043) 0:01:05.072 ***** 2020-12-05T19:02:09.915 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-12-05T19:02:09.917 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:09.952 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:09 +0000 (0:00:00.042) 0:01:05.115 ***** 2020-12-05T19:02:10.196 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-12-05T19:02:10.197 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-12-05T19:02:10.233 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.280) 0:01:05.396 ***** 2020-12-05T19:02:10.270 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-12-05T19:02:10.272 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-12-05T19:02:10.309 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.076) 0:01:05.472 ***** 2020-12-05T19:02:10.422 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-12-05T19:02:10.423 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-12-05T19:02:10.458 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.148) 0:01:05.621 ***** 2020-12-05T19:02:10.486 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-12-05T19:02:10.488 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.519 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.061) 0:01:05.682 ***** 2020-12-05T19:02:10.545 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.579 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.059) 0:01:05.742 ***** 2020-12-05T19:02:10.604 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-12-05T19:02:10.606 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.638 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.059) 0:01:05.801 ***** 2020-12-05T19:02:10.663 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-12-05T19:02:10.665 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-12-05T19:02:10.701 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.062) 0:01:05.864 ***** 2020-12-05T19:02:10.739 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.037) 0:01:05.902 ***** 2020-12-05T19:02:10.765 INFO:teuthology.task.ansible.out: 2020-12-05T19:02:10.767 INFO:teuthology.task.ansible.out:TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.813 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.074) 0:01:05.976 ***** 2020-12-05T19:02:10.845 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:10.847 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-12-05T19:02:10.887 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:10 +0000 (0:00:00.074) 0:01:06.050 ***** 2020-12-05T19:02:10.927 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-12-05T19:02:10.928 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.938 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.940 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.950 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.962 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.963 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.973 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-12-05T19:02:10.985 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.986 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.996 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:10.998 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.008 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.019 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.030 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.032 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.074 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.186) 0:01:06.237 ***** 2020-12-05T19:02:11.081 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-12-05T19:02:11.082 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-12-05T19:02:11.120 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.045) 0:01:06.282 ***** 2020-12-05T19:02:11.253 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-12-05T19:02:11.255 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003278", "end": "2020-12-05 19:02:11.231948", "rc": 0, "start": "2020-12-05 19:02:11.228670", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-12-05T19:02:11.302 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.182) 0:01:06.465 ***** 2020-12-05T19:02:11.353 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.050) 0:01:06.516 ***** 2020-12-05T19:02:11.360 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-12-05T19:02:11.362 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.400 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.047) 0:01:06.563 ***** 2020-12-05T19:02:11.406 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-12-05T19:02:11.408 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.446 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.045) 0:01:06.609 ***** 2020-12-05T19:02:11.452 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-12-05T19:02:11.454 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.489 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.043) 0:01:06.652 ***** 2020-12-05T19:02:11.495 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-12-05T19:02:11.497 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.531 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.042) 0:01:06.694 ***** 2020-12-05T19:02:11.537 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-12-05T19:02:11.538 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.576 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.044) 0:01:06.739 ***** 2020-12-05T19:02:11.582 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-12-05T19:02:11.584 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.618 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.041) 0:01:06.781 ***** 2020-12-05T19:02:11.624 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-12-05T19:02:11.625 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.661 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.043) 0:01:06.824 ***** 2020-12-05T19:02:11.688 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-12-05T19:02:11.690 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.724 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.062) 0:01:06.887 ***** 2020-12-05T19:02:11.761 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:11.763 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-12-05T19:02:11.796 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:11 +0000 (0:00:00.072) 0:01:06.960 ***** 2020-12-05T19:02:11.844 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-12-05T19:02:11.846 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.855 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.865 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.877 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.888 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.889 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.911 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-12-05T19:02:11.913 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.922 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.933 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.944 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.955 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.966 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.968 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:11.978 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:12.018 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.221) 0:01:07.181 ***** 2020-12-05T19:02:12.200 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-12-05T19:02:12.202 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003177", "end": "2020-12-05 19:02:12.180729", "rc": 0, "start": "2020-12-05 19:02:12.177552", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-12-05T19:02:12.238 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.220) 0:01:07.401 ***** 2020-12-05T19:02:12.264 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-12-05T19:02:12.266 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:12.301 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.062) 0:01:07.464 ***** 2020-12-05T19:02:12.326 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-12-05T19:02:12.328 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:12.364 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.063) 0:01:07.527 ***** 2020-12-05T19:02:12.401 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-12-05T19:02:12.403 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:12.439 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.074) 0:01:07.602 ***** 2020-12-05T19:02:12.507 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-12-05T19:02:12.509 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:02:12.545 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.106) 0:01:07.708 ***** 2020-12-05T19:02:12.572 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-12-05T19:02:12.574 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:12.614 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.068) 0:01:07.777 ***** 2020-12-05T19:02:12.651 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-12-05T19:02:12.653 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item={'block_used': 13755747, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226339361, 'size_available': 927086022656, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149527, 'block_size': 4096, 'inode_available': 60905449}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226339361, "block_size": 4096, "block_total": 240095108, "block_used": 13755747, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905449, "inode_total": 61054976, "inode_used": 149527, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927086022656, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}} 2020-12-05T19:02:12.689 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.074) 0:01:07.852 ***** 2020-12-05T19:02:12.830 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-12-05T19:02:12.832 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.008862", "end": "2020-12-05 19:02:12.809848", "rc": 0, "start": "2020-12-05 19:02:12.800986", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:12.866 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.176) 0:01:08.029 ***** 2020-12-05T19:02:12.902 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-12-05T19:02:12.904 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'block_used': 13755747, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226339361, 'size_available': 927086022656, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149527, 'block_size': 4096, 'inode_available': 60905449}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226339361, "block_size": 4096, "block_total": 240095108, "block_used": 13755747, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905449, "inode_total": 61054976, "inode_used": 149527, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927086022656, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:12.938 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:12 +0000 (0:00:00.072) 0:01:08.101 ***** 2020-12-05T19:02:12.966 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-12-05T19:02:13.002 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:13 +0000 (0:00:00.063) 0:01:08.165 ***** 2020-12-05T19:02:13.039 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-12-05T19:02:13.041 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:13.350 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-12-05T19:02:13.352 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:02:13.387 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:13 +0000 (0:00:00.385) 0:01:08.550 ***** 2020-12-05T19:02:13.414 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-12-05T19:02:13.416 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:13.450 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:13 +0000 (0:00:00.062) 0:01:08.613 ***** 2020-12-05T19:02:13.486 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-12-05T19:02:13.488 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'block_used': 13755746, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226339362, 'size_available': 927086026752, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149528, 'block_size': 4096, 'inode_available': 60905448}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226339362, "block_size": 4096, "block_total": 240095108, "block_used": 13755746, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905448, "inode_total": 61054976, "inode_used": 149528, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927086026752, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}} 2020-12-05T19:02:13.522 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:13 +0000 (0:00:00.071) 0:01:08.685 ***** 2020-12-05T19:02:13.651 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-12-05T19:02:13.653 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009752", "end": "2020-12-05 19:02:13.487040", "rc": 0, "start": "2020-12-05 19:02:13.477288", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:13.687 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:13 +0000 (0:00:00.164) 0:01:08.850 ***** 2020-12-05T19:02:13.723 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-12-05T19:02:13.725 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'block_used': 13755746, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226339362, 'size_available': 927086026752, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149528, 'block_size': 4096, 'inode_available': 60905448}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226339362, "block_size": 4096, "block_total": 240095108, "block_used": 13755746, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905448, "inode_total": 61054976, "inode_used": 149528, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927086026752, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:13.759 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:13 +0000 (0:00:00.072) 0:01:08.922 ***** 2020-12-05T19:02:13.785 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-12-05T19:02:13.819 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:13 +0000 (0:00:00.060) 0:01:08.982 ***** 2020-12-05T19:02:13.866 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-12-05T19:02:13.868 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:15.070 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJ81g2w7rcQna4xig1HCR90aTvAeRJHBd7'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.908624", "end": "2020-12-05 19:02:15.029600", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJ81g2w7rcQna4xig1HCR90aTvAeRJHBd7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:13.120976", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:15.071 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:15.082 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:15.879 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HgdMJ3QZZkbOL1J9Z9zBh1q0T7640WerL'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.887111", "end": "2020-12-05 19:02:15.700405", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HgdMJ3QZZkbOL1J9Z9zBh1q0T7640WerL"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:13.813294", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:15.891 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:15.891 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:16.337 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJcDJdSJLnfHgtK3nip4POW2OUHQoUho2j'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.120904", "end": "2020-12-05 19:02:16.293000", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJcDJdSJLnfHgtK3nip4POW2OUHQoUho2j"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:15.172096", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:16.338 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTPG', 'wwn-0x5000c5009293f2ac'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTPG-part1', 'wwn-0x5000c5009293f2ac-part1'], 'uuids': ['d8c38c2f-ff33-4bf4-853e-25622c47cb18']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293f2ac', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XTPG', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTPG", "wwn-0x5000c5009293f2ac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTPG-part1", "wwn-0x5000c5009293f2ac-part1"], "labels": [], "masters": [], "uuids": ["d8c38c2f-ff33-4bf4-853e-25622c47cb18"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTPG", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293f2ac"}}, "skip_reason": 2020-12-05T19:02:16.340 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-12-05T19:02:17.124 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HFj7Kv0HcUqSee4Pji7dJ27YnrioQF4MS'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.111018", "end": "2020-12-05 19:02:16.953651", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HFj7Kv0HcUqSee4Pji7dJ27YnrioQF4MS"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:15.842633", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:17.136 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3', 'wwn-0x5000c500871a974c'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'wwn-0x5000c500871a974c-part1'], 'uuids': ['d8c38c2f-ff33-4bf4-853e-25622c47cb18']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a974c', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HPS3', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3", "wwn-0x5000c500871a974c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1", "wwn-0x5000c500871a974c-part1"], "labels": [], "masters": [], "uuids": ["d8c38c2f-ff33-4bf4-853e-25622c47cb18"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPS3", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a974c"}}, "skip_reason": 2020-12-05T19:02:17.138 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-12-05T19:02:17.655 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-44w8Cz-dKb8-qR0U-nsiV-OTwr-lvhH-xEpeSv', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300KY400BGN', 'nvme-nvme.8086-435646543632333330304b5934303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.185079", "end": "2020-12-05 19:02:17.621547", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-44w8Cz-dKb8-qR0U-nsiV-OTwr-lvhH-xEpeSv", "nvme-INTEL_SSDPEDMD400G4_CVFT623300KY400BGN", "nvme-nvme.8086-435646543632333330304b5934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:16.436468", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-12-05T19:02:17.657 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:17.667 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:17.679 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:18.416 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '7814037168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-uBKOlY-tdVT-iesu-rQXM-kEsM-vKSw-voN6eE', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDKE040T7', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '3.64 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.148856", "end": "2020-12-05 19:02:18.238226", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600", "links": {"ids": ["lvm-pv-uuid-uBKOlY-tdVT-iesu-rQXM-kEsM-vKSw-voN6eE", "nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:17.089370", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT 2020-12-05T19:02:18.417 INFO:teuthology.task.ansible.out: data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:18.427 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:18.428 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:18.939 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJdBMXcUv7ZbgbVsct01QfmBGDAuGTidn3'], 'uuids': ['7d6fcabe-3f25-4078-9b1b-80155a532de5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.135712", "end": "2020-12-05 19:02:18.901773", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJdBMXcUv7ZbgbVsct01QfmBGDAuGTidn3"], "labels": [], "masters": [], "uuids": ["7d6fcabe-3f25-4078-9b1b-80155a532de5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:17.766061", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:18.951 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:18.952 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:19.689 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8H0YgjJjVyKAc5BuXkX56yzsYMTIq4HeFn'], 'uuids': ['c59dfa1b-8175-492b-9a12-b4a8f72df368']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.150360", "end": "2020-12-05 19:02:19.522705", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8H0YgjJjVyKAc5BuXkX56yzsYMTIq4HeFn"], "labels": [], "masters": [], "uuids": ["c59dfa1b-8175-492b-9a12-b4a8f72df368"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:18.372345", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:19.701 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:19.712 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:20.195 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJwhiBlG6DoKfx9AiXodDcjErC4jxKcGzU'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.128389", "end": "2020-12-05 19:02:20.159531", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJwhiBlG6DoKfx9AiXodDcjErC4jxKcGzU"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:19.031142", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:20.207 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:21.062 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HtC33PWzs4MT1PaEMmItN8EwqTx1EAKRj'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.243919", "end": "2020-12-05 19:02:20.898987", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HtC33PWzs4MT1PaEMmItN8EwqTx1EAKRj"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:19.655068", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:21.074 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:21.452 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJfNb76DvD7LNp0kpK7QYV3uaU4k0c05it'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.129026", "end": "2020-12-05 19:02:21.417965", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJfNb76DvD7LNp0kpK7QYV3uaU4k0c05it"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:20.288939", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:21.499 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:21 +0000 (0:00:07.679) 0:01:16.662 ***** 2020-12-05T19:02:21.547 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:21.820 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJ81g2w7rcQna4xig1HCR90aTvAeRJHBd7'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.154345", "end": "2020-12-05 19:02:21.782682", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJ81g2w7rcQna4xig1HCR90aTvAeRJHBd7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:21.628337", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149626 s, 771 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149626 s, 771 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:21.831 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:21.842 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:22.113 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJcDJdSJLnfHgtK3nip4POW2OUHQoUho2j'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.149551", "end": "2020-12-05 19:02:22.077445", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJcDJdSJLnfHgtK3nip4POW2OUHQoUho2j"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:21.927894", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146859 s, 785 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146859 s, 785 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:22.125 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTPG', 'wwn-0x5000c5009293f2ac'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTPG-part1', 'wwn-0x5000c5009293f2ac-part1'], 'uuids': ['d8c38c2f-ff33-4bf4-853e-25622c47cb18']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293f2ac', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XTPG', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTPG", "wwn-0x5000c5009293f2ac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTPG-part1", "wwn-0x5000c5009293f2ac-part1"], "labels": [], "masters": [], "uuids": ["d8c38c2f-ff33-4bf4-853e-25622c47cb18"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTPG", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293f2ac"}}, "skip_reason": 2020-12-05T19:02:22.128 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-12-05T19:02:22.304 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-44w8Cz-dKb8-qR0U-nsiV-OTwr-lvhH-xEpeSv', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300KY400BGN', 'nvme-nvme.8086-435646543632333330304b5934303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.046577", "end": "2020-12-05 19:02:22.261284", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-44w8Cz-dKb8-qR0U-nsiV-OTwr-lvhH-xEpeSv", "nvme-INTEL_SSDPEDMD400G4_CVFT623300KY400BGN", "nvme-nvme.8086-435646543632333330304b5934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:22.214707", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0437756 s, 2.6 GB/s", "std 2020-12-05T19:02:22.306 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0437756 s, 2.6 GB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:22.317 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:22.319 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:22.331 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-12-05T19:02:22.333 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HdcQGYVijPKVGZkkcLGYre2fjIErTBicO'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.131513", "end": "2020-12-05 19:02:22.157874", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HdcQGYVijPKVGZkkcLGYre2fjIErTBicO"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:21.026361", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-12-05T19:02:22.378 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:22 +0000 (0:00:00.879) 0:01:17.541 ***** 2020-12-05T19:02:22.426 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-12-05T19:02:22.428 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:22.595 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJdBMXcUv7ZbgbVsct01QfmBGDAuGTidn3'], 'uuids': ['7d6fcabe-3f25-4078-9b1b-80155a532de5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.147415", "end": "2020-12-05 19:02:22.555083", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJdBMXcUv7ZbgbVsct01QfmBGDAuGTidn3"], "labels": [], "masters": [], "uuids": ["7d6fcabe-3f25-4078-9b1b-80155a532de5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:22.407668", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144517 s, 798 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144517 s, 798 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:22.596 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:22.608 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:22.725 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HgdMJ3QZZkbOL1J9Z9zBh1q0T7640WerL'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.177657", "end": "2020-12-05 19:02:22.555647", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HgdMJ3QZZkbOL1J9Z9zBh1q0T7640WerL"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:22.377990", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.172844 s, 667 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.172844 s, 667 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:22.738 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:22.751 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:22.881 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJwhiBlG6DoKfx9AiXodDcjErC4jxKcGzU'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.154993", "end": "2020-12-05 19:02:22.843544", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJwhiBlG6DoKfx9AiXodDcjErC4jxKcGzU"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:22.688551", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15212 s, 758 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15212 s, 758 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:22.893 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:23.040 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HFj7Kv0HcUqSee4Pji7dJ27YnrioQF4MS'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.172406", "end": "2020-12-05 19:02:22.870975", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HFj7Kv0HcUqSee4Pji7dJ27YnrioQF4MS"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:22.698569", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.16764 s, 688 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.16764 s, 688 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:23.042 INFO:teuthology.task.ansible.out: 2020-12-05T19:02:23.055 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3', 'wwn-0x5000c500871a974c'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'wwn-0x5000c500871a974c-part1'], 'uuids': ['d8c38c2f-ff33-4bf4-853e-25622c47cb18']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a974c', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HPS3', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3", "wwn-0x5000c500871a974c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1", "wwn-0x5000c500871a974c-part1"], "labels": [], "masters": [], "uuids": ["d8c38c2f-ff33-4bf4-853e-25622c47cb18"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPS3", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a974c"}}, "skip_reason": 2020-12-05T19:02:23.056 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-12-05T19:02:23.162 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJfNb76DvD7LNp0kpK7QYV3uaU4k0c05it'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.152437", "end": "2020-12-05 19:02:23.118469", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EoQDBPd736zLJY5p3q0InQswKEPfB0IJfNb76DvD7LNp0kpK7QYV3uaU4k0c05it"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:22.966032", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149192 s, 773 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149192 s, 773 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:23.210 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:23 +0000 (0:00:00.831) 0:01:18.373 ***** 2020-12-05T19:02:23.228 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '7814037168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-uBKOlY-tdVT-iesu-rQXM-kEsM-vKSw-voN6eE', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDKE040T7', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '3.64 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.055646", "end": "2020-12-05 19:02:23.058045", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600", "links": {"ids": ["lvm-pv-uuid-uBKOlY-tdVT-iesu-rQXM-kEsM-vKSw-voN6eE", "nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:23.002399", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0507627 s, 2.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0507627 s, 2.3 GB 2020-12-05T19:02:23.230 INFO:teuthology.task.ansible.out:/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:23.239 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:23.251 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:23.480 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-12-05T19:02:23.482 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.144224", "end": "2020-12-05 19:02:23.449103", "rc": 0, "start": "2020-12-05 19:02:23.304879", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:23.515 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:23 +0000 (0:00:00.304) 0:01:18.678 ***** 2020-12-05T19:02:23.531 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-12-05T19:02:23.533 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8H0YgjJjVyKAc5BuXkX56yzsYMTIq4HeFn'], 'uuids': ['c59dfa1b-8175-492b-9a12-b4a8f72df368']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.165082", "end": "2020-12-05 19:02:23.356861", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8H0YgjJjVyKAc5BuXkX56yzsYMTIq4HeFn"], "labels": [], "masters": [], "uuids": ["c59dfa1b-8175-492b-9a12-b4a8f72df368"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:23.191779", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.162389 s, 710 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.162389 s, 710 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:23.543 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:23.545 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:23.681 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-12-05T19:02:23.683 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.042364", "end": "2020-12-05 19:02:23.649494", "rc": 0, "start": "2020-12-05 19:02:23.607130", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:23.718 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:23 +0000 (0:00:00.203) 0:01:18.881 ***** 2020-12-05T19:02:23.745 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-12-05T19:02:23.782 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:23 +0000 (0:00:00.063) 0:01:18.945 ***** 2020-12-05T19:02:23.808 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-12-05T19:02:23.810 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:23.848 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:23 +0000 (0:00:00.066) 0:01:19.011 ***** 2020-12-05T19:02:23.855 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-12-05T19:02:23.857 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HtC33PWzs4MT1PaEMmItN8EwqTx1EAKRj'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.172193", "end": "2020-12-05 19:02:23.675765", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HtC33PWzs4MT1PaEMmItN8EwqTx1EAKRj"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:23.503572", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.167527 s, 689 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.167527 s, 689 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:23.857 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": 2020-12-05T19:02:23.859 INFO:teuthology.task.ansible.out:{"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:23.868 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:23.905 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:23 +0000 (0:00:00.056) 0:01:19.068 ***** 2020-12-05T19:02:23.931 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-12-05T19:02:23.933 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:23.969 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:23 +0000 (0:00:00.064) 0:01:19.132 ***** 2020-12-05T19:02:23.995 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-12-05T19:02:23.997 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.033 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.064) 0:01:19.196 ***** 2020-12-05T19:02:24.135 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-12-05T19:02:24.137 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HdcQGYVijPKVGZkkcLGYre2fjIErTBicO'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.158929", "end": "2020-12-05 19:02:23.963573", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Wl2SQ00o3kHWrE2aNoygzdAk2Akwrf8HdcQGYVijPKVGZkkcLGYre2fjIErTBicO"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-12-05 19:02:23.804644", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154364 s, 747 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154364 s, 747 MB/s"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:24.182 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.148) 0:01:19.345 ***** 2020-12-05T19:02:24.189 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-12-05T19:02:24.190 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-12-05T19:02:24.226 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.043) 0:01:19.389 ***** 2020-12-05T19:02:24.262 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-12-05T19:02:24.264 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'block_used': 13755747, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226339361, 'size_available': 927086022656, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149527, 'block_size': 4096, 'inode_available': 60905449}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226339361, "block_size": 4096, "block_total": 240095108, "block_used": 13755747, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905449, "inode_total": 61054976, "inode_used": 149527, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927086022656, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.298 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.071) 0:01:19.461 ***** 2020-12-05T19:02:24.325 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-12-05T19:02:24.327 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.360 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.062) 0:01:19.523 ***** 2020-12-05T19:02:24.386 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:24.388 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.421 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.060) 0:01:19.584 ***** 2020-12-05T19:02:24.447 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-12-05T19:02:24.449 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.124133", "end": "2020-12-05 19:02:24.270148", "rc": 0, "start": "2020-12-05 19:02:24.146015", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : set_fact] ***************************************************** skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.482 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.061) 0:01:19.645 ***** 2020-12-05T19:02:24.515 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.033) 0:01:19.678 ***** 2020-12-05T19:02:24.552 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-12-05T19:02:24.554 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.592 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.076) 0:01:19.755 ***** 2020-12-05T19:02:24.618 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-12-05T19:02:24.620 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.654 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.061) 0:01:19.817 ***** 2020-12-05T19:02:24.660 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-12-05T19:02:24.662 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049573", "end": "2020-12-05 19:02:24.495233", "rc": 0, "start": "2020-12-05 19:02:24.445660", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:24.696 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.042) 0:01:19.859 ***** 2020-12-05T19:02:24.701 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-12-05T19:02:24.703 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.737 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.040) 0:01:19.900 ***** 2020-12-05T19:02:24.743 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-12-05T19:02:24.782 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.045) 0:01:19.945 ***** 2020-12-05T19:02:24.809 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-12-05T19:02:24.810 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.845 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.063) 0:01:20.008 ***** 2020-12-05T19:02:24.872 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-12-05T19:02:24.873 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.907 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.061) 0:01:20.070 ***** 2020-12-05T19:02:24.933 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-12-05T19:02:24.935 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:24.970 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:24 +0000 (0:00:00.062) 0:01:20.133 ***** 2020-12-05T19:02:24.996 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-12-05T19:02:24.998 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:25.030 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.060) 0:01:20.193 ***** 2020-12-05T19:02:25.160 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-12-05T19:02:25.162 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-12-05T19:02:25.211 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.180) 0:01:20.374 ***** 2020-12-05T19:02:25.251 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-12-05T19:02:25.252 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'block_used': 13755746, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226339362, 'size_available': 927086026752, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149528, 'block_size': 4096, 'inode_available': 60905448}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226339362, "block_size": 4096, "block_total": 240095108, "block_used": 13755746, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905448, "inode_total": 61054976, "inode_used": 149528, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927086026752, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:25.289 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.077) 0:01:20.452 ***** 2020-12-05T19:02:25.295 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-12-05T19:02:25.296 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-12-05T19:02:25.334 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.045) 0:01:20.497 ***** 2020-12-05T19:02:25.340 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-12-05T19:02:25.341 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:25.381 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.046) 0:01:20.544 ***** 2020-12-05T19:02:25.407 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:25.409 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:25.443 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.062) 0:01:20.606 ***** 2020-12-05T19:02:25.470 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:25.472 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:25.507 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.063) 0:01:20.670 ***** 2020-12-05T19:02:25.533 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-12-05T19:02:25.535 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:25.569 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.062) 0:01:20.732 ***** 2020-12-05T19:02:25.595 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-12-05T19:02:25.597 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:25.630 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.061) 0:01:20.793 ***** 2020-12-05T19:02:25.657 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-12-05T19:02:25.658 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:25.692 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:25 +0000 (0:00:00.061) 0:01:20.855 ***** 2020-12-05T19:02:25.864 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-12-05T19:02:25.866 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:26.011 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-12-05T19:02:26.012 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-12-05T19:02:26.046 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:26 +0000 (0:00:00.353) 0:01:21.209 ***** 2020-12-05T19:02:26.106 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-12-05T19:02:26.108 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:26.305 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:26.348 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:26.536 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:26.589 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:26.737 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:26.773 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:26 +0000 (0:00:00.727) 0:01:21.936 ***** 2020-12-05T19:02:26.822 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:26.895 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-12-05T19:02:26.897 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-12-05T19:02:26.931 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:26 +0000 (0:00:00.157) 0:01:22.094 ***** 2020-12-05T19:02:27.051 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-12-05T19:02:27.053 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:27.094 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-12-05T19:02:27.096 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-12-05T19:02:27.214 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-12-05T19:02:27.287 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-12-05T19:02:27.289 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-12-05T19:02:27.325 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:27 +0000 (0:00:00.394) 0:01:22.488 ***** 2020-12-05T19:02:27.332 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-12-05T19:02:27.334 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-12-05T19:02:27.448 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-12-05T19:02:27.459 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-12-05T19:02:27.461 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} TASK [testnode : Write /scratch_devs] ****************************************** skipping: [smithi185.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:27.499 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:27 +0000 (0:00:00.173) 0:01:22.662 ***** 2020-12-05T19:02:27.532 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:27 +0000 (0:00:00.033) 0:01:22.695 ***** 2020-12-05T19:02:27.559 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2020-12-05T19:02:27.561 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:27.595 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:27 +0000 (0:00:00.063) 0:01:22.758 ***** 2020-12-05T19:02:27.621 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2020-12-05T19:02:27.623 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:27.658 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:27 +0000 (0:00:00.062) 0:01:22.821 ***** 2020-12-05T19:02:27.664 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-12-05T19:02:27.665 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-12-05T19:02:27.749 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-12-05T19:02:27.792 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-12-05T19:02:27.793 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-12-05T19:02:27.827 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:27 +0000 (0:00:00.169) 0:01:22.991 ***** 2020-12-05T19:02:27.865 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-12-05T19:02:27.866 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-12-05T19:02:27.903 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:27 +0000 (0:00:00.075) 0:01:23.066 ***** 2020-12-05T19:02:27.971 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-12-05T19:02:27.983 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:28.020 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:28 +0000 (0:00:00.117) 0:01:23.183 ***** 2020-12-05T19:02:28.047 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2020-12-05T19:02:28.048 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:28.082 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:28 +0000 (0:00:00.061) 0:01:23.245 ***** 2020-12-05T19:02:28.108 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2020-12-05T19:02:28.109 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:28.143 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:28 +0000 (0:00:00.061) 0:01:23.306 ***** 2020-12-05T19:02:28.273 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-12-05T19:02:28.275 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-12-05T19:02:28.313 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:28 +0000 (0:00:00.169) 0:01:23.476 ***** 2020-12-05T19:02:28.350 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-12-05T19:02:28.352 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi185.front.sepia.ceph.com] => {"changed": true} 2020-12-05T19:02:28.387 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:28 +0000 (0:00:00.073) 0:01:23.550 ***** 2020-12-05T19:02:28.424 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:28 +0000 (0:00:00.037) 0:01:23.587 ***** 2020-12-05T19:02:28.773 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true} 2020-12-05T19:02:28.812 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:28 +0000 (0:00:00.387) 0:01:23.975 ***** 2020-12-05T19:02:28.870 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-12-05T19:02:28.872 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-12-05T19:02:28.907 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:28 +0000 (0:00:00.094) 0:01:24.070 ***** 2020-12-05T19:02:28.933 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2020-12-05T19:02:28.935 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:28.969 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:28 +0000 (0:00:00.061) 0:01:24.132 ***** 2020-12-05T19:02:28.995 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2020-12-05T19:02:28.997 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:29.031 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.062) 0:01:24.194 ***** 2020-12-05T19:02:29.037 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-12-05T19:02:29.039 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-12-05T19:02:29.075 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.043) 0:01:24.238 ***** 2020-12-05T19:02:29.081 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2020-12-05T19:02:29.083 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:29.117 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.041) 0:01:24.280 ***** 2020-12-05T19:02:29.122 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2020-12-05T19:02:29.124 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:29.159 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.042) 0:01:24.322 ***** 2020-12-05T19:02:29.165 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2020-12-05T19:02:29.167 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:29.203 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.043) 0:01:24.366 ***** 2020-12-05T19:02:29.209 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2020-12-05T19:02:29.210 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:29.247 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.043) 0:01:24.410 ***** 2020-12-05T19:02:29.273 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:29.312 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.065) 0:01:24.475 ***** 2020-12-05T19:02:29.338 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2020-12-05T19:02:29.340 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:29.373 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.061) 0:01:24.537 ***** 2020-12-05T19:02:29.380 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-12-05T19:02:29.382 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-12-05T19:02:29.419 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.045) 0:01:24.582 ***** 2020-12-05T19:02:29.520 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-12-05T19:02:29.557 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.137) 0:01:24.720 ***** 2020-12-05T19:02:29.563 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-12-05T19:02:29.565 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-12-05T19:02:29.601 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.043) 0:01:24.764 ***** 2020-12-05T19:02:29.690 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-12-05T19:02:29.726 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.125) 0:01:24.889 ***** 2020-12-05T19:02:29.911 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-12-05T19:02:29.912 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1235, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-12-05T19:02:29.963 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:29 +0000 (0:00:00.236) 0:01:25.126 ***** 2020-12-05T19:02:30.063 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1235, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-12-05T19:02:30.100 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:30 +0000 (0:00:00.137) 0:01:25.263 ***** 2020-12-05T19:02:30.273 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-12-05T19:02:30.274 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-12-05T19:02:30.311 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:30 +0000 (0:00:00.210) 0:01:25.474 ***** 2020-12-05T19:02:30.423 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-12-05T19:02:30.465 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:30 +0000 (0:00:00.153) 0:01:25.628 ***** 2020-12-05T19:02:31.193 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-12-05T19:02:31.195 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:02:31.252 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:31 +0000 (0:00:00.786) 0:01:26.415 ***** 2020-12-05T19:02:31.409 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:02:31.449 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:31 +0000 (0:00:00.196) 0:01:26.611 ***** 2020-12-05T19:02:31.456 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-12-05T19:02:31.458 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.044688", "end": "2020-12-05 19:02:31.391599", "rc": 0, "start": "2020-12-05 19:02:31.346911", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-12-05T19:02:31.494 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:31 +0000 (0:00:00.045) 0:01:26.657 ***** 2020-12-05T19:02:31.521 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-12-05T19:02:31.522 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:31.558 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:31 +0000 (0:00:00.063) 0:01:26.721 ***** 2020-12-05T19:02:31.584 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-12-05T19:02:31.585 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:31.623 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:31 +0000 (0:00:00.065) 0:01:26.786 ***** 2020-12-05T19:02:31.630 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-12-05T19:02:31.632 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.052115", "end": "2020-12-05 19:02:31.469162", "rc": 0, "start": "2020-12-05 19:02:31.417047", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-12-05T19:02:31.672 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:31 +0000 (0:00:00.048) 0:01:26.835 ***** 2020-12-05T19:02:31.698 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-12-05T19:02:31.700 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:31.736 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:31 +0000 (0:00:00.064) 0:01:26.899 ***** 2020-12-05T19:02:31.764 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-12-05T19:02:31.766 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:31.801 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:31 +0000 (0:00:00.065) 0:01:26.964 ***** 2020-12-05T19:02:32.473 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-12-05T19:02:32.475 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:02:32.509 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:32 +0000 (0:00:00.707) 0:01:27.672 ***** 2020-12-05T19:02:32.536 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-12-05T19:02:32.537 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:32.582 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:32 +0000 (0:00:00.073) 0:01:27.745 ***** 2020-12-05T19:02:32.697 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-12-05T19:02:32.699 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:02:32.735 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:32 +0000 (0:00:00.152) 0:01:27.898 ***** 2020-12-05T19:02:32.762 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-12-05T19:02:32.764 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:32.798 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:32 +0000 (0:00:00.063) 0:01:27.961 ***** 2020-12-05T19:02:32.882 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-12-05T19:02:32.884 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-12-05T19:02:32.919 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:32 +0000 (0:00:00.121) 0:01:28.082 ***** 2020-12-05T19:02:32.946 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-12-05T19:02:32.947 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:32.982 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:32 +0000 (0:00:00.062) 0:01:28.144 ***** 2020-12-05T19:02:33.101 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-12-05T19:02:33.103 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-12-05T19:02:33.137 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.155) 0:01:28.300 ***** 2020-12-05T19:02:33.163 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-12-05T19:02:33.165 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:33.199 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.062) 0:01:28.362 ***** 2020-12-05T19:02:33.206 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-12-05T19:02:33.207 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2020-12-05 19:00:41 UTC", "ActiveEnterTimestampMonotonic": "24988081", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target sysinit.target network.target systemd-journald.socket systemd-tmpfiles-setup.service -.mount system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2020-12-05 19:00:35 UTC", "AssertTimestampMonotonic": "18480074", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2020-12-05 19:00:35 UTC", "ConditionTimestampMonotonic": "18480074", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-12-05T19:02:33.209 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2003", "ExecMainStartTimestamp": "Sat 2020-12-05 19:00:41 UTC", "ExecMainStartTimestampMonotonic": "24988067", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2020-12-05 19:00:35 UTC", "InactiveExitTimestampMonotonic": "18480891", "InvocationID": "94ed328ac4d748939254062587ae216a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-12-05T19:02:33.210 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2003", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2020-12-05 19:00:41 UTC", "S 2020-12-05T19:02:33.212 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24988081", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2020-12-05 19:00:41 UTC", "WatchdogTimestampMonotonic": "24988080", "WatchdogUSec": "0"}} 2020-12-05T19:02:33.244 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.044) 0:01:28.407 ***** 2020-12-05T19:02:33.269 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:33.271 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-12-05T19:02:33.304 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.060) 0:01:28.467 ***** 2020-12-05T19:02:33.329 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:33.331 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:33.363 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.058) 0:01:28.526 ***** 2020-12-05T19:02:33.387 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-12-05T19:02:33.389 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:33.420 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.057) 0:01:28.583 ***** 2020-12-05T19:02:33.426 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-12-05T19:02:33.427 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2020-12-05 19:01:02 UTC", "ActiveEnterTimestampMonotonic": "26613876", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-tmpfiles-setup.service basic.target system.slice sysinit.target systemd-journald.socket network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2020-12-05 19:00:55 UTC", "AssertTimestampMonotonic": "20048040", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2020-12-05 19:00:55 UTC", "ConditionTimestampMonotonic": "20048040", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-12-05T19:02:33.429 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1908", "ExecMainStartTimestamp": "Sat 2020-12-05 19:01:02 UTC", "ExecMainStartTimestampMonotonic": "26613846", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2020-12-05 19:00:55 UTC", "InactiveExitTimestampMonotonic": "20048736", "InvocationID": "0d446feb9a744285847690ae1e4e4610", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-12-05T19:02:33.430 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1908", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2020-12-05 19:01:02 UTC", "S 2020-12-05T19:02:33.432 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "26613876", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2020-12-05 19:01:02 UTC", "WatchdogTimestampMonotonic": "26613874", "WatchdogUSec": "0"}} 2020-12-05T19:02:33.463 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.042) 0:01:28.626 ***** 2020-12-05T19:02:33.479 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:33.481 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-12-05T19:02:33.516 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.053) 0:01:28.679 ***** 2020-12-05T19:02:33.541 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:33.574 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.058) 0:01:28.737 ***** 2020-12-05T19:02:33.600 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-12-05T19:02:33.601 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:33.635 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:33 +0000 (0:00:00.060) 0:01:28.798 ***** 2020-12-05T19:02:34.239 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-12-05T19:02:34.241 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:02:34.276 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:34 +0000 (0:00:00.641) 0:01:29.439 ***** 2020-12-05T19:02:34.303 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-12-05T19:02:34.305 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:34.339 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:34 +0000 (0:00:00.062) 0:01:29.502 ***** 2020-12-05T19:02:34.458 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-12-05T19:02:34.460 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-12-05T19:02:34.494 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:34 +0000 (0:00:00.154) 0:01:29.657 ***** 2020-12-05T19:02:34.531 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-12-05T19:02:34.533 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:02:34.568 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:34 +0000 (0:00:00.073) 0:01:29.731 ***** 2020-12-05T19:02:34.594 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-12-05T19:02:34.595 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:34.629 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:34 +0000 (0:00:00.061) 0:01:29.792 ***** 2020-12-05T19:02:34.749 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-12-05T19:02:34.750 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-12-05T19:02:34.785 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:34 +0000 (0:00:00.156) 0:01:29.948 ***** 2020-12-05T19:02:34.790 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-12-05T19:02:34.792 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1235, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-12-05T19:02:34.827 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:34 +0000 (0:00:00.042) 0:01:29.990 ***** 2020-12-05T19:02:35.093 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1235, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-12-05T19:02:35.132 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:35 +0000 (0:00:00.304) 0:01:30.295 ***** 2020-12-05T19:02:35.857 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-12-05T19:02:35.859 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-12-05T19:02:35.895 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:35 +0000 (0:00:00.763) 0:01:31.058 ***** 2020-12-05T19:02:35.901 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-12-05T19:02:35.942 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:35 +0000 (0:00:00.046) 0:01:31.104 ***** 2020-12-05T19:02:36.394 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-12-05T19:02:36.396 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false} 2020-12-05T19:02:36.431 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:36 +0000 (0:00:00.489) 0:01:31.594 ***** 2020-12-05T19:02:36.458 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-12-05T19:02:36.460 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-12-05T19:02:36.492 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:36 +0000 (0:00:00.060) 0:01:31.655 ***** 2020-12-05T19:02:36.497 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-12-05T19:02:36.499 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false} 2020-12-05T19:02:36.536 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:36 +0000 (0:00:00.044) 0:01:31.699 ***** 2020-12-05T19:02:36.564 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-12-05T19:02:36.566 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-12-05T19:02:36.599 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:36 +0000 (0:00:00.063) 0:01:31.762 ***** 2020-12-05T19:02:36.627 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-12-05T19:02:36.629 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1607194596.8180141, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1607194596.8140163, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 113087, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1607194596.8140163, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3531524185", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-12-05T19:02:36.665 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:36 +0000 (0:00:00.065) 0:01:31.827 ***** 2020-12-05T19:02:36.734 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1607194614.5737834, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1607194614.5697834, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 113087, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1607194614.5697834, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3531524185", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-12-05T19:02:36.777 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:36 +0000 (0:00:00.112) 0:01:31.940 ***** 2020-12-05T19:02:37.015 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-12-05T19:02:37.017 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-12-05T19:02:37.053 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.276) 0:01:32.216 ***** 2020-12-05T19:02:37.087 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.033) 0:01:32.250 ***** 2020-12-05T19:02:37.093 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-12-05T19:02:37.095 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-12-05T19:02:37.132 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.045) 0:01:32.295 ***** 2020-12-05T19:02:37.139 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-12-05T19:02:37.177 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.044) 0:01:32.340 ***** 2020-12-05T19:02:37.182 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-12-05T19:02:37.184 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.219 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.042) 0:01:32.382 ***** 2020-12-05T19:02:37.225 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.262 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.042) 0:01:32.425 ***** 2020-12-05T19:02:37.340 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-12-05T19:02:37.342 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-12-05T19:02:37.378 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.116) 0:01:32.541 ***** 2020-12-05T19:02:37.395 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-12-05T19:02:37.436 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.057) 0:01:32.599 ***** 2020-12-05T19:02:37.505 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-12-05T19:02:37.507 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-12-05T19:02:37.541 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.105) 0:01:32.704 ***** 2020-12-05T19:02:37.568 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-12-05T19:02:37.570 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.604 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.062) 0:01:32.767 ***** 2020-12-05T19:02:37.637 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.033) 0:01:32.800 ***** 2020-12-05T19:02:37.643 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-12-05T19:02:37.644 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.646 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.658 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.660 INFO:teuthology.task.ansible.out: 2020-12-05T19:02:37.661 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.714 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.077) 0:01:32.877 ***** 2020-12-05T19:02:37.747 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.032) 0:01:32.910 ***** 2020-12-05T19:02:37.765 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-12-05T19:02:37.766 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.767 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.778 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.780 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.820 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.072) 0:01:32.983 ***** 2020-12-05T19:02:37.826 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-12-05T19:02:37.828 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.829 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.865 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.045) 0:01:33.028 ***** 2020-12-05T19:02:37.871 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:37.873 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.912 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.046) 0:01:33.075 ***** 2020-12-05T19:02:37.919 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-12-05T19:02:37.920 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi185.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.921 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.921 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditiona 2020-12-05T19:02:37.923 INFO:teuthology.task.ansible.out:l result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:37.959 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:37 +0000 (0:00:00.047) 0:01:33.122 ***** 2020-12-05T19:02:37.965 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-12-05T19:02:37.970 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.003 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.044) 0:01:33.166 ***** 2020-12-05T19:02:38.011 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-12-05T19:02:38.013 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.050 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.047) 0:01:33.213 ***** 2020-12-05T19:02:38.056 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-12-05T19:02:38.058 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.095 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.044) 0:01:33.258 ***** 2020-12-05T19:02:38.102 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-12-05T19:02:38.104 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.142 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.047) 0:01:33.305 ***** 2020-12-05T19:02:38.150 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-12-05T19:02:38.152 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.188 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.045) 0:01:33.351 ***** 2020-12-05T19:02:38.194 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-12-05T19:02:38.196 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.233 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.045) 0:01:33.396 ***** 2020-12-05T19:02:38.239 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-12-05T19:02:38.241 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.278 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.044) 0:01:33.441 ***** 2020-12-05T19:02:38.285 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-12-05T19:02:38.287 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.323 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.045) 0:01:33.486 ***** 2020-12-05T19:02:38.330 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-12-05T19:02:38.332 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.375 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.051) 0:01:33.538 ***** 2020-12-05T19:02:38.381 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-12-05T19:02:38.383 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.422 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.046) 0:01:33.585 ***** 2020-12-05T19:02:38.429 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-12-05T19:02:38.430 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.469 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.046) 0:01:33.632 ***** 2020-12-05T19:02:38.476 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-12-05T19:02:38.478 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.516 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.047) 0:01:33.679 ***** 2020-12-05T19:02:38.523 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-12-05T19:02:38.524 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.560 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.043) 0:01:33.723 ***** 2020-12-05T19:02:38.567 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-12-05T19:02:38.568 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.607 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.046) 0:01:33.770 ***** 2020-12-05T19:02:38.633 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-12-05T19:02:38.635 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:38.669 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:38 +0000 (0:00:00.062) 0:01:33.832 ***** 2020-12-05T19:02:38.718 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-12-05T19:02:38.720 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010925", "end": "2020-12-05 19:02:38.545092", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-12-05 19:02:38.534167", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:38.823 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011149", "end": "2020-12-05 19:02:38.776220", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-12-05 19:02:38.765071", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:38.824 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005640", "end": "2020-12-05 19:02:38.665949", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-12-05 19:02:38.660309", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:38.939 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006045", "end": "2020-12-05 19:02:38.895034", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-12-05 19:02:38.888989", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:38.941 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005482", "end": "2020-12-05 19:02:38.780172", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-12-05 19:02:38.774690", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:39.047 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005461", "end": "2020-12-05 19:02:39.010666", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-12-05 19:02:39.005205", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:39.079 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.024272", "end": "2020-12-05 19:02:38.910767", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-12-05 19:02:38.886495", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:39.119 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:39 +0000 (0:00:00.449) 0:01:34.282 ***** 2020-12-05T19:02:39.145 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-12-05T19:02:39.183 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:39 +0000 (0:00:00.063) 0:01:34.345 ***** 2020-12-05T19:02:39.188 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-12-05T19:02:39.189 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026269", "end": "2020-12-05 19:02:39.141353", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-12-05 19:02:39.115084", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:39.231 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:39 +0000 (0:00:00.047) 0:01:34.393 ***** 2020-12-05T19:02:39.257 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-12-05T19:02:39.299 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:39 +0000 (0:00:00.068) 0:01:34.462 ***** 2020-12-05T19:02:39.305 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-12-05T19:02:39.307 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-12-05T19:02:39.455 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-12-05T19:02:39.582 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi] *************************************************** 2020-12-05T19:02:39.617 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:39 +0000 (0:00:00.317) 0:01:34.780 ***** 2020-12-05T19:02:39.653 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:39 +0000 (0:00:00.035) 0:01:34.815 ***** 2020-12-05T19:02:40.378 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi023.front.sepia.ceph.com] 2020-12-05T19:02:40.425 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] 2020-12-05T19:02:40.466 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:40 +0000 (0:00:00.813) 0:01:35.629 ***** 2020-12-05T19:02:40.535 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:40 +0000 (0:00:00.069) 0:01:35.698 ***** 2020-12-05T19:02:40.541 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2020-12-05T19:02:40.543 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2020-12-05T19:02:40.579 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:40 +0000 (0:00:00.043) 0:01:35.742 ***** 2020-12-05T19:02:40.586 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2020-12-05T19:02:40.622 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:40 +0000 (0:00:00.042) 0:01:35.785 ***** 2020-12-05T19:02:40.628 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2020-12-05T19:02:40.630 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:40.665 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:40 +0000 (0:00:00.043) 0:01:35.828 ***** 2020-12-05T19:02:40.671 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:40.710 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:40 +0000 (0:00:00.044) 0:01:35.873 ***** 2020-12-05T19:02:40.715 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2020-12-05T19:02:40.717 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2020-12-05T19:02:40.754 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:40 +0000 (0:00:00.044) 0:01:35.917 ***** 2020-12-05T19:02:40.759 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2020-12-05T19:02:40.797 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:40 +0000 (0:00:00.043) 0:01:35.960 ***** 2020-12-05T19:02:41.680 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2020-12-05T19:02:41.682 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"cache_update_time": 1607194681, "cache_updated": false, "changed": false} 2020-12-05T19:02:41.726 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:41 +0000 (0:00:00.928) 0:01:36.889 ***** 2020-12-05T19:02:41.735 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"cache_update_time": 1607194637, "cache_updated": false, "changed": false} 2020-12-05T19:02:41.773 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:41 +0000 (0:00:00.046) 0:01:36.936 ***** 2020-12-05T19:02:41.780 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2020-12-05T19:02:41.782 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:41.819 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:41 +0000 (0:00:00.046) 0:01:36.982 ***** 2020-12-05T19:02:41.825 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:41.865 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:41 +0000 (0:00:00.046) 0:01:37.028 ***** 2020-12-05T19:02:41.871 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2020-12-05T19:02:41.873 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2020-12-05T19:02:41.911 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:41 +0000 (0:00:00.046) 0:01:37.074 ***** 2020-12-05T19:02:41.918 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2020-12-05T19:02:41.956 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:41 +0000 (0:00:00.044) 0:01:37.119 ***** 2020-12-05T19:02:42.046 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000] *** 2020-12-05T19:02:42.047 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2020-12-05T19:02:42.087 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:42 +0000 (0:00:00.131) 0:01:37.250 ***** 2020-12-05T19:02:42.094 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2020-12-05T19:02:42.132 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:42 +0000 (0:00:00.045) 0:01:37.295 ***** 2020-12-05T19:02:42.409 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2020-12-05T19:02:42.411 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1607194962.1153083-109794659051278/source", "state": "file", "uid": 0} 2020-12-05T19:02:42.447 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:42 +0000 (0:00:00.314) 0:01:37.610 ***** 2020-12-05T19:02:42.454 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1607194962.1596878-85458267835163/source", "state": "file", "uid": 0} 2020-12-05T19:02:42.491 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:42 +0000 (0:00:00.044) 0:01:37.654 ***** 2020-12-05T19:02:45.957 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2020-12-05T19:02:45.958 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-JjtKKW-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-JjtKKW-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2020-12-05T19:02:45.960 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2020-12-05T19:02:45.995 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:45 +0000 (0:00:03.503) 0:01:41.158 ***** 2020-12-05T19:02:46.002 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6Q7drV-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6Q7drV-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2020-12-05T19:02:46.003 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2020-12-05T19:02:46.042 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.047) 0:01:41.205 ***** 2020-12-05T19:02:46.121 INFO:teuthology.task.ansible.out: TASK [container-host : Find registries-conf-ctl] ******************************* 2020-12-05T19:02:46.123 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-12-05T19:02:46.169 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.127) 0:01:41.332 ***** 2020-12-05T19:02:46.176 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-12-05T19:02:46.216 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.046) 0:01:41.379 ***** 2020-12-05T19:02:46.308 INFO:teuthology.task.ansible.out: 2020-12-05T19:02:46.310 INFO:teuthology.task.ansible.out:TASK [container-host : Find registries-conf-ctl again] ************************* ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1607194965.3415236, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "checksum": "7c9342d5a9fce575259687fd2510a929c13d7ab7", "ctime": 1607194965.3415236, "dev": 2049, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 66903, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/x-python", "mode": "0755", "mtime": 1607194965.3415236, "nlink": 1, "path": "/usr/local/bin/registries-conf-ctl", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 435, "uid": 0, "version": "3418765867", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}} 2020-12-05T19:02:46.348 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.132) 0:01:41.511 ***** 2020-12-05T19:02:46.365 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1607194965.180396, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "checksum": "7c9342d5a9fce575259687fd2510a929c13d7ab7", "ctime": 1607194965.180396, "dev": 2049, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 66903, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/x-python", "mode": "0755", "mtime": 1607194965.180396, "nlink": 1, "path": "/usr/local/bin/registries-conf-ctl", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 435, "uid": 0, "version": "2069258717", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}} 2020-12-05T19:02:46.404 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.056) 0:01:41.567 ***** 2020-12-05T19:02:46.410 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2020-12-05T19:02:46.412 INFO:teuthology.task.ansible.out:skipping: [smithi185.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:46.451 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.046) 0:01:41.614 ***** 2020-12-05T19:02:46.460 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-12-05T19:02:46.501 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.050) 0:01:41.664 ***** 2020-12-05T19:02:46.507 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2020-12-05T19:02:46.509 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"ansible_facts": {"rcc_path": "/usr/local/bin/registries-conf-ctl"}, "changed": false} 2020-12-05T19:02:46.546 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.044) 0:01:41.709 ***** 2020-12-05T19:02:46.551 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"rcc_path": "/usr/local/bin/registries-conf-ctl"}, "changed": false} 2020-12-05T19:02:46.602 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.056) 0:01:41.765 ***** 2020-12-05T19:02:46.673 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2020-12-05T19:02:46.674 INFO:teuthology.task.ansible.out:ok: [smithi185.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-12-05T19:02:46.713 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.110) 0:01:41.875 ***** 2020-12-05T19:02:46.740 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-12-05T19:02:46.777 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:46 +0000 (0:00:00.064) 0:01:41.940 ***** 2020-12-05T19:02:47.033 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2020-12-05T19:02:47.035 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1607194966.7454643-137859314777404/source", "state": "file", "uid": 0} 2020-12-05T19:02:47.074 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:47 +0000 (0:00:00.296) 0:01:42.237 ***** 2020-12-05T19:02:47.132 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1607194966.8105123-254519003250360/source", "state": "file", "uid": 0} 2020-12-05T19:02:47.167 INFO:teuthology.task.ansible.out:Saturday 05 December 2020 19:02:47 +0000 (0:00:00.093) 0:01:42.331 ***** 2020-12-05T19:02:47.319 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2020-12-05T19:02:47.321 INFO:teuthology.task.ansible.out:changed: [smithi185.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/local/bin/registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.113175", "end": "2020-12-05 19:02:47.288374", "rc": 0, "start": "2020-12-05 19:02:47.175199", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:47.425 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/local/bin/registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.123686", "end": "2020-12-05 19:02:47.273801", "rc": 0, "start": "2020-12-05 19:02:47.150115", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-12-05T19:02:47.521 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-12-05T19:02:47.742 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi023.front.sepia.ceph.com : ok=122 changed=32 unreachable=0 failed=0 skipped=190 rescued=0 ignored=0 smithi185.front.sepia.ceph.com : ok=122 changed=32 unreachable=0 failed=0 skipped=190 rescued=0 ignored=0 Saturday 05 December 2020 19:02:47 +0000 (0:00:00.360) 0:01:42.691 ***** =============================================================================== testnode : Ensure packages are not present. ----------------------------- 8.39s testnode : Install packages --------------------------------------------- 7.84s testnode : Zap all non-root disks --------------------------------------- 7.68s common : Update apt cache ----------------------------------------------- 3.90s container-host : Install registries-conf-ctl ---------------------------- 3.50s testnode : Update apt cache. -------------------------------------------- 2.83s testnode : Install apt keys --------------------------------------------- 1.86s Gathering Facts --------------------------------------------------------- 1.81s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 0.95s container-host : Install container packages ----------------------------- 0.93s common : Install nrpe package and dependencies (Ubuntu/Fedora) ---------- 0.90s common : Make sure smartmontools is installed --------------------------- 0.85s users : Create all admin users with sudo access. ------------------------ 0.84s testnode : Install python-apt ------------------------------------------- 0.84s testnode : Blow away lingering OSD data and FSIDs ----------------------- 0.83s common : Upload custom netsaint scripts for raid/disk 2020-12-05T19:02:47.743 INFO:teuthology.task.ansible.out:/smart/monitoring to /usr/libexec/. --- 0.83s testnode : Set the system's hostname ------------------------------------ 0.81s Gathering Facts --------------------------------------------------------- 0.81s common : Set the hardware clock ----------------------------------------- 0.81s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.79s 2020-12-05T19:02:47.846 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi185.front.sepia.ceph.com'), Remote(name='ubuntu@smithi023.front.sepia.ceph.com')] 2020-12-05T19:02:47.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:02:47.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:02:47.982 INFO:teuthology.orchestra.run.smithi185:> true 2020-12-05T19:02:48.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-12-05T19:02:48.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-12-05T19:02:48.254 INFO:teuthology.orchestra.run.smithi023:> true 2020-12-05T19:02:48.446 INFO:teuthology.run_tasks:Running task clock... 2020-12-05T19:02:48.458 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-12-05T19:02:48.459 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-12-05T19:02:48.459 INFO:teuthology.orchestra.run.smithi185:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-05T19:02:48.516 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-12-05T19:02:48.517 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: Command line: ntpd -gq 2020-12-05T19:02:48.518 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: proto: precision = 0.124 usec (-23) 2020-12-05T19:02:48.519 INFO:teuthology.orchestra.run.smithi185.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-12-05T19:02:48.519 INFO:teuthology.orchestra.run.smithi185.stderr:restrict ::: KOD does nothing without LIMITED. 2020-12-05T19:02:48.519 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-12-05T19:02:48.520 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: restrict ::: KOD does nothing without LIMITED. 2020-12-05T19:02:48.520 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: Listen and drop on 0 v6wildcard [::]:123 2020-12-05T19:02:48.520 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-12-05T19:02:48.520 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: Listen normally on 2 lo 127.0.0.1:123 2020-12-05T19:02:48.520 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: Listen normally on 3 enp3s0f1 172.21.15.185:123 2020-12-05T19:02:48.521 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: Listen normally on 4 lo [::1]:123 2020-12-05T19:02:48.521 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e49%5]:123 2020-12-05T19:02:48.521 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:48 ntpd[6164]: Listening on routing socket on fd #22 for interface updates 2020-12-05T19:02:49.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:49 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:49.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:49 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:49.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:49 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:49.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:49 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:49.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:49 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:49.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:49 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:49.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:49 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:49.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:49 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:50.517 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:50 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:50.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:50 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:50.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:50 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:50.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:50 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:51.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:51 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:51.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:51 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:51.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:51 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:51.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:51 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:51.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:51 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:51.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:51 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:51.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:51 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:51.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:51 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:52.517 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:52 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:52.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:52 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:52.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:52 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:52.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:52 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:52.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:52 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:52.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:52 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:52.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:52 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:52.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:52 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:53.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:53 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:53.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:53 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:53.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:53 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:53.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:53 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:53.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:53 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:53.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:53 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:53.520 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:53 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:53.520 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:53 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:54.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:54 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:54.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:54 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:54.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:54 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:54.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:54 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:54.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:54 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:54.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:54 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:54.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:54 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:54.520 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:54 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:55.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:55 ntpd[6164]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:55.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:55 ntpd[6164]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:55.518 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:55 ntpd[6164]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:55.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:55 ntpd[6164]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:55.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:55 ntpd[6164]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-12-05T19:02:55.519 INFO:teuthology.orchestra.run.smithi185.stderr: 5 Dec 19:02:55 ntpd[6164]: can't open /var/log/ntpstats/loopstats.20201205: Permission denied 2020-12-05T19:02:55.519 INFO:teuthology.orchestra.run.smithi185.stdout: 5 Dec 19:02:55 ntpd[6164]: ntpd: time slew +0.016797 s 2020-12-05T19:02:55.520 INFO:teuthology.orchestra.run.smithi185.stdout:ntpd: time slew +0.016797s 2020-12-05T19:02:55.573 INFO:teuthology.orchestra.run.smithi185.stdout: remote refid st t when poll reach delay offset jitter 2020-12-05T19:02:55.573 INFO:teuthology.orchestra.run.smithi185.stdout:============================================================================== 2020-12-05T19:02:55.573 INFO:teuthology.orchestra.run.smithi185.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-12-05T19:02:55.573 INFO:teuthology.orchestra.run.smithi185.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-12-05T19:02:55.574 INFO:teuthology.orchestra.run.smithi185.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-12-05T19:02:55.574 INFO:teuthology.orchestra.run.smithi185.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-12-05T19:02:55.574 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-12-05T19:02:55.575 INFO:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-05T19:02:55.636 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-12-05T19:02:55.637 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: Command line: ntpd -gq 2020-12-05T19:02:55.638 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: proto: precision = 0.126 usec (-23) 2020-12-05T19:02:55.639 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-12-05T19:02:55.639 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: restrict ::: KOD does nothing without LIMITED. 2020-12-05T19:02:55.639 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: Listen and drop on 0 v6wildcard [::]:123 2020-12-05T19:02:55.639 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-12-05T19:02:55.640 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: Listen normally on 2 lo 127.0.0.1:123 2020-12-05T19:02:55.640 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: Listen normally on 3 ens1f0 172.21.15.23:123 2020-12-05T19:02:55.640 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: Listen normally on 4 lo [::1]:123 2020-12-05T19:02:55.640 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe59:ce42%3]:123 2020-12-05T19:02:55.640 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:02:55 ntpd[6018]: Listening on routing socket on fd #22 for interface updates 2020-12-05T19:02:55.641 INFO:teuthology.orchestra.run.smithi023.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-12-05T19:02:55.641 INFO:teuthology.orchestra.run.smithi023.stderr:restrict ::: KOD does nothing without LIMITED. 2020-12-05T19:02:56.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:56 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:56.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:56 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:56.638 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:56 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:56.638 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:56 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:57.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:57 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:57.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:57 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:57.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:57 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:57.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:57 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:58.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:58 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:58.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:58 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:58.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:58 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:58.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:58 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:58.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:58 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:58.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:58 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:58.638 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:58 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:58.638 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:58 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:59.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:59 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:59.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:59 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:59.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:59 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:59.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:59 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:02:59.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:59 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:02:59.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:59 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:02:59.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:59 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:02:59.638 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:02:59 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:03:00.635 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:00 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:03:00.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:00 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:03:00.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:00 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:03:00.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:00 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:03:00.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:00 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:03:00.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:00 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:03:00.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:00 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:03:00.637 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:00 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:03:01.635 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:01 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:03:01.635 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:01 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:03:01.635 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:01 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:03:01.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:01 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:03:01.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:01 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:03:01.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:01 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:03:01.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:01 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:03:01.636 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:01 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:03:02.634 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:02 ntpd[6018]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-12-05T19:03:02.634 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:02 ntpd[6018]: can't open /var/log/ntpstats/rawstats.20201205: Permission denied 2020-12-05T19:03:02.634 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:02 ntpd[6018]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-12-05T19:03:02.635 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:02 ntpd[6018]: can't open /var/log/ntpstats/peerstats.20201205: Permission denied 2020-12-05T19:03:02.635 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:02 ntpd[6018]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-12-05T19:03:02.635 INFO:teuthology.orchestra.run.smithi023.stderr: 5 Dec 19:03:02 ntpd[6018]: can't open /var/log/ntpstats/loopstats.20201205: Permission denied 2020-12-05T19:03:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: 5 Dec 19:03:02 ntpd[6018]: ntpd: time set +0.126985 s 2020-12-05T19:03:02.635 INFO:teuthology.orchestra.run.smithi023.stdout:ntpd: time set +0.126985s 2020-12-05T19:03:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: remote refid st t when poll reach delay offset jitter 2020-12-05T19:03:02.680 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================== 2020-12-05T19:03:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-12-05T19:03:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-12-05T19:03:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-12-05T19:03:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-12-05T19:03:02.682 INFO:teuthology.run_tasks:Running task install... 2020-12-05T19:03:02.693 DEBUG:teuthology.task.install:project ceph 2020-12-05T19:03:02.693 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674'}} 2020-12-05T19:03:02.693 DEBUG:teuthology.task.install:config {'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674'} 2020-12-05T19:03:02.694 INFO:teuthology.task.install:Using flavor: basic 2020-12-05T19:03:02.700 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-12-05T19:03:02.700 INFO:teuthology.task.install:extra packages: [] 2020-12-05T19:03:02.701 INFO:teuthology.orchestra.run.smithi185:> sudo lsb_release -is 2020-12-05T19:03:02.746 INFO:teuthology.orchestra.run.smithi185.stdout:Ubuntu 2020-12-05T19:03:02.751 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-12-05T19:03:02.751 INFO:teuthology.orchestra.run.smithi023:> sudo lsb_release -is 2020-12-05T19:03:02.752 INFO:teuthology.orchestra.run.smithi185:> sudo apt-key list | grep Ceph 2020-12-05T19:03:02.807 INFO:teuthology.orchestra.run.smithi023.stdout:Ubuntu 2020-12-05T19:03:02.813 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-12-05T19:03:02.813 INFO:teuthology.orchestra.run.smithi023:> sudo apt-key list | grep Ceph 2020-12-05T19:03:02.886 INFO:teuthology.orchestra.run.smithi185.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-12-05T19:03:02.906 INFO:teuthology.orchestra.run.smithi023.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-12-05T19:03:02.940 INFO:teuthology.orchestra.run.smithi185.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-12-05T19:03:02.940 INFO:teuthology.orchestra.run.smithi185.stdout:uid [ unknown] Ceph.com (release key) 2020-12-05T19:03:02.941 INFO:teuthology.orchestra.run.smithi185:> sudo lsb_release -is 2020-12-05T19:03:02.964 INFO:teuthology.orchestra.run.smithi023.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-12-05T19:03:02.964 INFO:teuthology.orchestra.run.smithi023.stdout:uid [ unknown] Ceph.com (release key) 2020-12-05T19:03:02.965 INFO:teuthology.orchestra.run.smithi023:> sudo lsb_release -is 2020-12-05T19:03:02.990 INFO:teuthology.orchestra.run.smithi185.stdout:Ubuntu 2020-12-05T19:03:02.995 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-12-05T19:03:02.995 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-12-05T19:03:02.996 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3b1861acb8137d863184795ccadbd5d0cb026674 2020-12-05T19:03:03.018 INFO:teuthology.orchestra.run.smithi023.stdout:Ubuntu 2020-12-05T19:03:03.030 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-12-05T19:03:03.030 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-12-05T19:03:03.030 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3b1861acb8137d863184795ccadbd5d0cb026674 2020-12-05T19:03:03.158 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default/ 2020-12-05T19:03:03.158 INFO:teuthology.task.install.deb:Package version is 15.2.6-607-g3b1861ac-1bionic 2020-12-05T19:03:03.198 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default/ 2020-12-05T19:03:03.198 INFO:teuthology.task.install.deb:Package version is 15.2.6-607-g3b1861ac-1bionic 2020-12-05T19:03:03.316 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:03:03.317 INFO:teuthology.orchestra.run.smithi185:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-12-05T19:03:03.321 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:03:03.322 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-12-05T19:03:03.322 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-12-05T19:03:03.327 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-12-05T19:03:03.330 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records in 2020-12-05T19:03:03.331 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records out 2020-12-05T19:03:03.331 INFO:teuthology.orchestra.run.smithi185.stderr:143 bytes copied, 4.8676e-05 s, 2.9 MB/s 2020-12-05T19:03:03.332 INFO:teuthology.orchestra.run.smithi185:> sudo apt-get update 2020-12-05T19:03:03.336 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-12-05T19:03:03.336 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-12-05T19:03:03.337 INFO:teuthology.orchestra.run.smithi023.stderr:143 bytes copied, 7.734e-05 s, 1.8 MB/s 2020-12-05T19:03:03.337 INFO:teuthology.orchestra.run.smithi023:> sudo apt-get update 2020-12-05T19:03:03.576 INFO:teuthology.orchestra.run.smithi185.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic InRelease 2020-12-05T19:03:03.591 INFO:teuthology.orchestra.run.smithi023.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic InRelease 2020-12-05T19:03:03.600 INFO:teuthology.orchestra.run.smithi185.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-12-05T19:03:03.615 INFO:teuthology.orchestra.run.smithi023.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-12-05T19:03:03.632 INFO:teuthology.orchestra.run.smithi185.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-12-05T19:03:03.632 INFO:teuthology.orchestra.run.smithi185.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-12-05T19:03:03.634 INFO:teuthology.orchestra.run.smithi185.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic Release.gpg 2020-12-05T19:03:03.650 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-12-05T19:03:03.650 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-12-05T19:03:03.652 INFO:teuthology.orchestra.run.smithi023.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic Release.gpg 2020-12-05T19:03:03.658 INFO:teuthology.orchestra.run.smithi185.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2020-12-05T19:03:03.675 INFO:teuthology.orchestra.run.smithi023.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2020-12-05T19:03:03.696 INFO:teuthology.orchestra.run.smithi185.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,081 B] 2020-12-05T19:03:03.706 INFO:teuthology.orchestra.run.smithi185.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-12-05T19:03:03.717 INFO:teuthology.orchestra.run.smithi023.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,081 B] 2020-12-05T19:03:03.718 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-12-05T19:03:03.787 INFO:teuthology.orchestra.run.smithi185.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-12-05T19:03:03.797 INFO:teuthology.orchestra.run.smithi023.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-12-05T19:03:04.244 INFO:teuthology.orchestra.run.smithi185.stdout:Fetched 26.6 kB in 1s (37.3 kB/s) 2020-12-05T19:03:04.271 INFO:teuthology.orchestra.run.smithi023.stdout:Fetched 26.6 kB in 1s (36.6 kB/s) 2020-12-05T19:03:05.472 INFO:teuthology.orchestra.run.smithi185.stdout:Reading package lists... 2020-12-05T19:03:05.521 INFO:teuthology.orchestra.run.smithi185:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.6-607-g3b1861ac-1bionic cephadm=15.2.6-607-g3b1861ac-1bionic ceph-mds=15.2.6-607-g3b1861ac-1bionic ceph-mgr=15.2.6-607-g3b1861ac-1bionic ceph-common=15.2.6-607-g3b1861ac-1bionic ceph-fuse=15.2.6-607-g3b1861ac-1bionic ceph-test=15.2.6-607-g3b1861ac-1bionic radosgw=15.2.6-607-g3b1861ac-1bionic python3-rados=15.2.6-607-g3b1861ac-1bionic python3-rgw=15.2.6-607-g3b1861ac-1bionic python3-cephfs=15.2.6-607-g3b1861ac-1bionic python3-rbd=15.2.6-607-g3b1861ac-1bionic libcephfs2=15.2.6-607-g3b1861ac-1bionic libcephfs-dev=15.2.6-607-g3b1861ac-1bionic librados2=15.2.6-607-g3b1861ac-1bionic librbd1=15.2.6-607-g3b1861ac-1bionic rbd-fuse=15.2.6-607-g3b1861ac-1bionic 2020-12-05T19:03:05.555 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2020-12-05T19:03:05.602 INFO:teuthology.orchestra.run.smithi023:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.6-607-g3b1861ac-1bionic cephadm=15.2.6-607-g3b1861ac-1bionic ceph-mds=15.2.6-607-g3b1861ac-1bionic ceph-mgr=15.2.6-607-g3b1861ac-1bionic ceph-common=15.2.6-607-g3b1861ac-1bionic ceph-fuse=15.2.6-607-g3b1861ac-1bionic ceph-test=15.2.6-607-g3b1861ac-1bionic radosgw=15.2.6-607-g3b1861ac-1bionic python3-rados=15.2.6-607-g3b1861ac-1bionic python3-rgw=15.2.6-607-g3b1861ac-1bionic python3-cephfs=15.2.6-607-g3b1861ac-1bionic python3-rbd=15.2.6-607-g3b1861ac-1bionic libcephfs2=15.2.6-607-g3b1861ac-1bionic libcephfs-dev=15.2.6-607-g3b1861ac-1bionic librados2=15.2.6-607-g3b1861ac-1bionic librbd1=15.2.6-607-g3b1861ac-1bionic rbd-fuse=15.2.6-607-g3b1861ac-1bionic 2020-12-05T19:03:05.605 INFO:teuthology.orchestra.run.smithi185.stdout:Reading package lists... 2020-12-05T19:03:05.685 INFO:teuthology.orchestra.run.smithi023.stdout:Reading package lists... 2020-12-05T19:03:05.864 INFO:teuthology.orchestra.run.smithi185.stdout:Building dependency tree... 2020-12-05T19:03:05.865 INFO:teuthology.orchestra.run.smithi185.stdout:Reading state information... 2020-12-05T19:03:05.922 INFO:teuthology.orchestra.run.smithi023.stdout:Building dependency tree... 2020-12-05T19:03:05.923 INFO:teuthology.orchestra.run.smithi023.stdout:Reading state information... 2020-12-05T19:03:06.031 INFO:teuthology.orchestra.run.smithi185.stdout:The following additional packages will be installed: 2020-12-05T19:03:06.031 INFO:teuthology.orchestra.run.smithi185.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-12-05T19:03:06.032 INFO:teuthology.orchestra.run.smithi185.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-12-05T19:03:06.033 INFO:teuthology.orchestra.run.smithi185.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-12-05T19:03:06.034 INFO:teuthology.orchestra.run.smithi185.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-12-05T19:03:06.034 INFO:teuthology.orchestra.run.smithi185.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2020-12-05T19:03:06.034 INFO:teuthology.orchestra.run.smithi185.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-12-05T19:03:06.034 INFO:teuthology.orchestra.run.smithi185.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-12-05T19:03:06.034 INFO:teuthology.orchestra.run.smithi185.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-12-05T19:03:06.035 INFO:teuthology.orchestra.run.smithi185.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-12-05T19:03:06.035 INFO:teuthology.orchestra.run.smithi185.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-12-05T19:03:06.035 INFO:teuthology.orchestra.run.smithi185.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-12-05T19:03:06.035 INFO:teuthology.orchestra.run.smithi185.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-12-05T19:03:06.035 INFO:teuthology.orchestra.run.smithi185.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-12-05T19:03:06.036 INFO:teuthology.orchestra.run.smithi185.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-12-05T19:03:06.036 INFO:teuthology.orchestra.run.smithi185.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-12-05T19:03:06.036 INFO:teuthology.orchestra.run.smithi185.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-12-05T19:03:06.036 INFO:teuthology.orchestra.run.smithi185.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-12-05T19:03:06.036 INFO:teuthology.orchestra.run.smithi185.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-12-05T19:03:06.037 INFO:teuthology.orchestra.run.smithi185.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-12-05T19:03:06.037 INFO:teuthology.orchestra.run.smithi185.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-12-05T19:03:06.037 INFO:teuthology.orchestra.run.smithi185.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-12-05T19:03:06.037 INFO:teuthology.orchestra.run.smithi185.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-12-05T19:03:06.037 INFO:teuthology.orchestra.run.smithi185.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-12-05T19:03:06.038 INFO:teuthology.orchestra.run.smithi185.stdout:Suggested packages: 2020-12-05T19:03:06.039 INFO:teuthology.orchestra.run.smithi185.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-12-05T19:03:06.039 INFO:teuthology.orchestra.run.smithi185.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-12-05T19:03:06.039 INFO:teuthology.orchestra.run.smithi185.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-12-05T19:03:06.039 INFO:teuthology.orchestra.run.smithi185.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-12-05T19:03:06.040 INFO:teuthology.orchestra.run.smithi185.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-12-05T19:03:06.040 INFO:teuthology.orchestra.run.smithi185.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-12-05T19:03:06.040 INFO:teuthology.orchestra.run.smithi185.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-12-05T19:03:06.040 INFO:teuthology.orchestra.run.smithi185.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-12-05T19:03:06.041 INFO:teuthology.orchestra.run.smithi185.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-12-05T19:03:06.041 INFO:teuthology.orchestra.run.smithi185.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-12-05T19:03:06.041 INFO:teuthology.orchestra.run.smithi185.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-12-05T19:03:06.041 INFO:teuthology.orchestra.run.smithi185.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-12-05T19:03:06.042 INFO:teuthology.orchestra.run.smithi185.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-12-05T19:03:06.042 INFO:teuthology.orchestra.run.smithi185.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-12-05T19:03:06.042 INFO:teuthology.orchestra.run.smithi185.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-12-05T19:03:06.099 INFO:teuthology.orchestra.run.smithi023.stdout:The following additional packages will be installed: 2020-12-05T19:03:06.100 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-12-05T19:03:06.100 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-12-05T19:03:06.101 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-12-05T19:03:06.101 INFO:teuthology.orchestra.run.smithi023.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-12-05T19:03:06.101 INFO:teuthology.orchestra.run.smithi023.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2020-12-05T19:03:06.102 INFO:teuthology.orchestra.run.smithi023.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-12-05T19:03:06.102 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-12-05T19:03:06.102 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-12-05T19:03:06.102 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-12-05T19:03:06.102 INFO:teuthology.orchestra.run.smithi023.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-12-05T19:03:06.103 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-12-05T19:03:06.103 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-12-05T19:03:06.103 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-12-05T19:03:06.103 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-12-05T19:03:06.103 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-12-05T19:03:06.104 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-12-05T19:03:06.104 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-12-05T19:03:06.104 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-12-05T19:03:06.104 INFO:teuthology.orchestra.run.smithi023.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-12-05T19:03:06.104 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-12-05T19:03:06.105 INFO:teuthology.orchestra.run.smithi023.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-12-05T19:03:06.105 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-12-05T19:03:06.105 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-12-05T19:03:06.105 INFO:teuthology.orchestra.run.smithi023.stdout:Suggested packages: 2020-12-05T19:03:06.106 INFO:teuthology.orchestra.run.smithi023.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-12-05T19:03:06.106 INFO:teuthology.orchestra.run.smithi023.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-12-05T19:03:06.106 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-12-05T19:03:06.106 INFO:teuthology.orchestra.run.smithi023.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-12-05T19:03:06.107 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-12-05T19:03:06.107 INFO:teuthology.orchestra.run.smithi023.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-12-05T19:03:06.107 INFO:teuthology.orchestra.run.smithi023.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-12-05T19:03:06.107 INFO:teuthology.orchestra.run.smithi023.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-12-05T19:03:06.107 INFO:teuthology.orchestra.run.smithi023.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-12-05T19:03:06.108 INFO:teuthology.orchestra.run.smithi023.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-12-05T19:03:06.108 INFO:teuthology.orchestra.run.smithi023.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-12-05T19:03:06.108 INFO:teuthology.orchestra.run.smithi023.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-12-05T19:03:06.108 INFO:teuthology.orchestra.run.smithi023.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-12-05T19:03:06.108 INFO:teuthology.orchestra.run.smithi023.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-12-05T19:03:06.109 INFO:teuthology.orchestra.run.smithi023.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-12-05T19:03:06.137 INFO:teuthology.orchestra.run.smithi185.stdout:The following NEW packages will be installed: 2020-12-05T19:03:06.137 INFO:teuthology.orchestra.run.smithi185.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-12-05T19:03:06.138 INFO:teuthology.orchestra.run.smithi185.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-12-05T19:03:06.138 INFO:teuthology.orchestra.run.smithi185.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-12-05T19:03:06.140 INFO:teuthology.orchestra.run.smithi185.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-12-05T19:03:06.140 INFO:teuthology.orchestra.run.smithi185.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-12-05T19:03:06.140 INFO:teuthology.orchestra.run.smithi185.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-12-05T19:03:06.140 INFO:teuthology.orchestra.run.smithi185.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-12-05T19:03:06.141 INFO:teuthology.orchestra.run.smithi185.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-12-05T19:03:06.141 INFO:teuthology.orchestra.run.smithi185.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-12-05T19:03:06.141 INFO:teuthology.orchestra.run.smithi185.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-12-05T19:03:06.141 INFO:teuthology.orchestra.run.smithi185.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-12-05T19:03:06.141 INFO:teuthology.orchestra.run.smithi185.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-12-05T19:03:06.142 INFO:teuthology.orchestra.run.smithi185.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-12-05T19:03:06.142 INFO:teuthology.orchestra.run.smithi185.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-12-05T19:03:06.142 INFO:teuthology.orchestra.run.smithi185.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-12-05T19:03:06.142 INFO:teuthology.orchestra.run.smithi185.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-12-05T19:03:06.142 INFO:teuthology.orchestra.run.smithi185.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-12-05T19:03:06.143 INFO:teuthology.orchestra.run.smithi185.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-12-05T19:03:06.143 INFO:teuthology.orchestra.run.smithi185.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-12-05T19:03:06.143 INFO:teuthology.orchestra.run.smithi185.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-12-05T19:03:06.143 INFO:teuthology.orchestra.run.smithi185.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-12-05T19:03:06.143 INFO:teuthology.orchestra.run.smithi185.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-12-05T19:03:06.144 INFO:teuthology.orchestra.run.smithi185.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-12-05T19:03:06.144 INFO:teuthology.orchestra.run.smithi185.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-12-05T19:03:06.144 INFO:teuthology.orchestra.run.smithi185.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-12-05T19:03:06.145 INFO:teuthology.orchestra.run.smithi185.stdout:The following packages will be upgraded: 2020-12-05T19:03:06.145 INFO:teuthology.orchestra.run.smithi185.stdout: librados2 librbd1 2020-12-05T19:03:06.214 INFO:teuthology.orchestra.run.smithi023.stdout:The following NEW packages will be installed: 2020-12-05T19:03:06.214 INFO:teuthology.orchestra.run.smithi023.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-12-05T19:03:06.215 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-12-05T19:03:06.215 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-12-05T19:03:06.215 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-12-05T19:03:06.216 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-12-05T19:03:06.216 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-12-05T19:03:06.217 INFO:teuthology.orchestra.run.smithi023.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-12-05T19:03:06.217 INFO:teuthology.orchestra.run.smithi023.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-12-05T19:03:06.217 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-12-05T19:03:06.217 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-12-05T19:03:06.217 INFO:teuthology.orchestra.run.smithi023.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-12-05T19:03:06.218 INFO:teuthology.orchestra.run.smithi023.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-12-05T19:03:06.218 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-12-05T19:03:06.218 INFO:teuthology.orchestra.run.smithi023.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-12-05T19:03:06.218 INFO:teuthology.orchestra.run.smithi023.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-12-05T19:03:06.218 INFO:teuthology.orchestra.run.smithi023.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-12-05T19:03:06.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-12-05T19:03:06.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-12-05T19:03:06.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-12-05T19:03:06.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-12-05T19:03:06.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-12-05T19:03:06.220 INFO:teuthology.orchestra.run.smithi023.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-12-05T19:03:06.220 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-12-05T19:03:06.220 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-12-05T19:03:06.220 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-12-05T19:03:06.221 INFO:teuthology.orchestra.run.smithi023.stdout:The following packages will be upgraded: 2020-12-05T19:03:06.221 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 librbd1 2020-12-05T19:03:06.303 INFO:teuthology.orchestra.run.smithi185.stdout:2 upgraded, 111 newly installed, 0 to remove and 3 not upgraded. 2020-12-05T19:03:06.304 INFO:teuthology.orchestra.run.smithi185.stdout:Need to get 156 MB of archives. 2020-12-05T19:03:06.304 INFO:teuthology.orchestra.run.smithi185.stdout:After this operation, 724 MB of additional disk space will be used. 2020-12-05T19:03:06.304 INFO:teuthology.orchestra.run.smithi185.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.6-607-g3b1861ac-1bionic [1,634 kB] 2020-12-05T19:03:06.331 INFO:teuthology.orchestra.run.smithi185.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-12-05T19:03:06.380 INFO:teuthology.orchestra.run.smithi023.stdout:2 upgraded, 111 newly installed, 0 to remove and 3 not upgraded. 2020-12-05T19:03:06.380 INFO:teuthology.orchestra.run.smithi023.stdout:Need to get 156 MB of archives. 2020-12-05T19:03:06.380 INFO:teuthology.orchestra.run.smithi023.stdout:After this operation, 724 MB of additional disk space will be used. 2020-12-05T19:03:06.381 INFO:teuthology.orchestra.run.smithi023.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.6-607-g3b1861ac-1bionic [1,634 kB] 2020-12-05T19:03:06.401 INFO:teuthology.orchestra.run.smithi023.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-12-05T19:03:06.413 INFO:teuthology.orchestra.run.smithi185.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-12-05T19:03:06.481 INFO:teuthology.orchestra.run.smithi023.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-12-05T19:03:06.514 INFO:teuthology.orchestra.run.smithi185.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.6-607-g3b1861ac-1bionic [3,059 kB] 2020-12-05T19:03:06.590 INFO:teuthology.orchestra.run.smithi023.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.6-607-g3b1861ac-1bionic [3,059 kB] 2020-12-05T19:03:06.670 INFO:teuthology.orchestra.run.smithi185.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-12-05T19:03:06.682 INFO:teuthology.orchestra.run.smithi185.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-12-05T19:03:06.689 INFO:teuthology.orchestra.run.smithi185.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.6-607-g3b1861ac-1bionic [435 kB] 2020-12-05T19:03:06.698 INFO:teuthology.orchestra.run.smithi185.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-12-05T19:03:06.716 INFO:teuthology.orchestra.run.smithi185.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.6-607-g3b1861ac-1bionic [332 kB] 2020-12-05T19:03:06.735 INFO:teuthology.orchestra.run.smithi023.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-12-05T19:03:06.737 INFO:teuthology.orchestra.run.smithi185.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.6-607-g3b1861ac-1bionic [20.7 kB] 2020-12-05T19:03:06.738 INFO:teuthology.orchestra.run.smithi185.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.6-607-g3b1861ac-1bionic [133 kB] 2020-12-05T19:03:06.740 INFO:teuthology.orchestra.run.smithi023.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-12-05T19:03:06.745 INFO:teuthology.orchestra.run.smithi185.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.6-607-g3b1861ac-1bionic [27.8 kB] 2020-12-05T19:03:06.746 INFO:teuthology.orchestra.run.smithi023.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-12-05T19:03:06.747 INFO:teuthology.orchestra.run.smithi185.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.6-607-g3b1861ac-1bionic [312 kB] 2020-12-05T19:03:06.768 INFO:teuthology.orchestra.run.smithi185.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.6-607-g3b1861ac-1bionic [2,889 kB] 2020-12-05T19:03:06.797 INFO:teuthology.orchestra.run.smithi185.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-12-05T19:03:06.800 INFO:teuthology.orchestra.run.smithi185.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-12-05T19:03:06.801 INFO:teuthology.orchestra.run.smithi023.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.6-607-g3b1861ac-1bionic [435 kB] 2020-12-05T19:03:06.821 INFO:teuthology.orchestra.run.smithi023.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.6-607-g3b1861ac-1bionic [332 kB] 2020-12-05T19:03:06.828 INFO:teuthology.orchestra.run.smithi185.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-12-05T19:03:06.836 INFO:teuthology.orchestra.run.smithi185.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-12-05T19:03:06.844 INFO:teuthology.orchestra.run.smithi185.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-12-05T19:03:06.851 INFO:teuthology.orchestra.run.smithi023.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.6-607-g3b1861ac-1bionic [20.7 kB] 2020-12-05T19:03:06.852 INFO:teuthology.orchestra.run.smithi023.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.6-607-g3b1861ac-1bionic [133 kB] 2020-12-05T19:03:06.853 INFO:teuthology.orchestra.run.smithi023.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-12-05T19:03:06.855 INFO:teuthology.orchestra.run.smithi185.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-12-05T19:03:06.856 INFO:teuthology.orchestra.run.smithi023.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.6-607-g3b1861ac-1bionic [27.8 kB] 2020-12-05T19:03:06.857 INFO:teuthology.orchestra.run.smithi023.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-12-05T19:03:06.864 INFO:teuthology.orchestra.run.smithi185.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-12-05T19:03:06.870 INFO:teuthology.orchestra.run.smithi023.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.6-607-g3b1861ac-1bionic [312 kB] 2020-12-05T19:03:06.887 INFO:teuthology.orchestra.run.smithi023.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.6-607-g3b1861ac-1bionic [2,889 kB] 2020-12-05T19:03:06.887 INFO:teuthology.orchestra.run.smithi023.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-12-05T19:03:06.888 INFO:teuthology.orchestra.run.smithi185.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-12-05T19:03:06.891 INFO:teuthology.orchestra.run.smithi185.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-12-05T19:03:06.895 INFO:teuthology.orchestra.run.smithi023.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-12-05T19:03:06.903 INFO:teuthology.orchestra.run.smithi023.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-12-05T19:03:06.913 INFO:teuthology.orchestra.run.smithi185.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-12-05T19:03:06.914 INFO:teuthology.orchestra.run.smithi023.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-12-05T19:03:06.924 INFO:teuthology.orchestra.run.smithi185.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-12-05T19:03:06.924 INFO:teuthology.orchestra.run.smithi023.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-12-05T19:03:06.926 INFO:teuthology.orchestra.run.smithi185.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-12-05T19:03:06.944 INFO:teuthology.orchestra.run.smithi185.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.6-607-g3b1861ac-1bionic [99.3 kB] 2020-12-05T19:03:06.947 INFO:teuthology.orchestra.run.smithi185.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.6-607-g3b1861ac-1bionic [337 kB] 2020-12-05T19:03:06.952 INFO:teuthology.orchestra.run.smithi023.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-12-05T19:03:06.955 INFO:teuthology.orchestra.run.smithi023.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-12-05T19:03:06.972 INFO:teuthology.orchestra.run.smithi023.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-12-05T19:03:06.976 INFO:teuthology.orchestra.run.smithi185.stdout:Get:29 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.6-607-g3b1861ac-1bionic [18.8 MB] 2020-12-05T19:03:06.983 INFO:teuthology.orchestra.run.smithi023.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-12-05T19:03:06.984 INFO:teuthology.orchestra.run.smithi023.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-12-05T19:03:06.994 INFO:teuthology.orchestra.run.smithi185.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-12-05T19:03:06.995 INFO:teuthology.orchestra.run.smithi185.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-12-05T19:03:06.997 INFO:teuthology.orchestra.run.smithi185.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-12-05T19:03:06.998 INFO:teuthology.orchestra.run.smithi185.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-12-05T19:03:06.998 INFO:teuthology.orchestra.run.smithi185.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-12-05T19:03:07.001 INFO:teuthology.orchestra.run.smithi185.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-12-05T19:03:07.004 INFO:teuthology.orchestra.run.smithi185.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-12-05T19:03:07.050 INFO:teuthology.orchestra.run.smithi023.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-12-05T19:03:07.051 INFO:teuthology.orchestra.run.smithi023.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-12-05T19:03:07.053 INFO:teuthology.orchestra.run.smithi023.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-12-05T19:03:07.053 INFO:teuthology.orchestra.run.smithi023.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-12-05T19:03:07.054 INFO:teuthology.orchestra.run.smithi023.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-12-05T19:03:07.057 INFO:teuthology.orchestra.run.smithi023.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-12-05T19:03:07.059 INFO:teuthology.orchestra.run.smithi023.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-12-05T19:03:07.075 INFO:teuthology.orchestra.run.smithi185.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-12-05T19:03:07.075 INFO:teuthology.orchestra.run.smithi185.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-12-05T19:03:07.095 INFO:teuthology.orchestra.run.smithi023.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.6-607-g3b1861ac-1bionic [99.3 kB] 2020-12-05T19:03:07.096 INFO:teuthology.orchestra.run.smithi185.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-12-05T19:03:07.097 INFO:teuthology.orchestra.run.smithi023.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.6-607-g3b1861ac-1bionic [337 kB] 2020-12-05T19:03:07.128 INFO:teuthology.orchestra.run.smithi023.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-12-05T19:03:07.129 INFO:teuthology.orchestra.run.smithi023.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-12-05T19:03:07.135 INFO:teuthology.orchestra.run.smithi023.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.6-607-g3b1861ac-1bionic [18.8 MB] 2020-12-05T19:03:07.149 INFO:teuthology.orchestra.run.smithi023.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-12-05T19:03:07.155 INFO:teuthology.orchestra.run.smithi185.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-12-05T19:03:07.156 INFO:teuthology.orchestra.run.smithi185.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-12-05T19:03:07.157 INFO:teuthology.orchestra.run.smithi185.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-12-05T19:03:07.160 INFO:teuthology.orchestra.run.smithi185.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2020-12-05T19:03:07.169 INFO:teuthology.orchestra.run.smithi185.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-12-05T19:03:07.174 INFO:teuthology.orchestra.run.smithi185.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-12-05T19:03:07.175 INFO:teuthology.orchestra.run.smithi185.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-12-05T19:03:07.206 INFO:teuthology.orchestra.run.smithi023.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-12-05T19:03:07.207 INFO:teuthology.orchestra.run.smithi023.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-12-05T19:03:07.208 INFO:teuthology.orchestra.run.smithi023.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-12-05T19:03:07.212 INFO:teuthology.orchestra.run.smithi023.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2020-12-05T19:03:07.220 INFO:teuthology.orchestra.run.smithi023.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-12-05T19:03:07.224 INFO:teuthology.orchestra.run.smithi023.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-12-05T19:03:07.225 INFO:teuthology.orchestra.run.smithi023.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-12-05T19:03:07.236 INFO:teuthology.orchestra.run.smithi185.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-12-05T19:03:07.284 INFO:teuthology.orchestra.run.smithi023.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-12-05T19:03:07.325 INFO:teuthology.orchestra.run.smithi185.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-12-05T19:03:07.326 INFO:teuthology.orchestra.run.smithi185.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-12-05T19:03:07.379 INFO:teuthology.orchestra.run.smithi023.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-12-05T19:03:07.379 INFO:teuthology.orchestra.run.smithi023.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-12-05T19:03:07.609 INFO:teuthology.orchestra.run.smithi185.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-12-05T19:03:07.649 INFO:teuthology.orchestra.run.smithi023.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-12-05T19:03:07.653 INFO:teuthology.orchestra.run.smithi185.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-12-05T19:03:07.654 INFO:teuthology.orchestra.run.smithi185.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-12-05T19:03:07.689 INFO:teuthology.orchestra.run.smithi185.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-12-05T19:03:07.689 INFO:teuthology.orchestra.run.smithi185.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-12-05T19:03:07.690 INFO:teuthology.orchestra.run.smithi185.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-12-05T19:03:07.690 INFO:teuthology.orchestra.run.smithi185.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-12-05T19:03:07.690 INFO:teuthology.orchestra.run.smithi185.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-12-05T19:03:07.691 INFO:teuthology.orchestra.run.smithi185.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-12-05T19:03:07.691 INFO:teuthology.orchestra.run.smithi185.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-12-05T19:03:07.704 INFO:teuthology.orchestra.run.smithi023.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-12-05T19:03:07.706 INFO:teuthology.orchestra.run.smithi023.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-12-05T19:03:07.728 INFO:teuthology.orchestra.run.smithi023.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-12-05T19:03:07.729 INFO:teuthology.orchestra.run.smithi023.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-12-05T19:03:07.729 INFO:teuthology.orchestra.run.smithi023.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-12-05T19:03:07.729 INFO:teuthology.orchestra.run.smithi023.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-12-05T19:03:07.730 INFO:teuthology.orchestra.run.smithi023.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-12-05T19:03:07.731 INFO:teuthology.orchestra.run.smithi023.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-12-05T19:03:07.731 INFO:teuthology.orchestra.run.smithi023.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-12-05T19:03:07.771 INFO:teuthology.orchestra.run.smithi185.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-12-05T19:03:07.773 INFO:teuthology.orchestra.run.smithi185.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-12-05T19:03:07.780 INFO:teuthology.orchestra.run.smithi185.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-12-05T19:03:07.781 INFO:teuthology.orchestra.run.smithi185.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-12-05T19:03:07.785 INFO:teuthology.orchestra.run.smithi185.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-12-05T19:03:07.806 INFO:teuthology.orchestra.run.smithi023.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-12-05T19:03:07.808 INFO:teuthology.orchestra.run.smithi023.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-12-05T19:03:07.851 INFO:teuthology.orchestra.run.smithi185.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-12-05T19:03:07.852 INFO:teuthology.orchestra.run.smithi185.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-12-05T19:03:07.856 INFO:teuthology.orchestra.run.smithi185.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-12-05T19:03:07.859 INFO:teuthology.orchestra.run.smithi185.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-12-05T19:03:07.861 INFO:teuthology.orchestra.run.smithi185.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-12-05T19:03:07.884 INFO:teuthology.orchestra.run.smithi023.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-12-05T19:03:07.885 INFO:teuthology.orchestra.run.smithi023.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-12-05T19:03:07.891 INFO:teuthology.orchestra.run.smithi023.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-12-05T19:03:07.892 INFO:teuthology.orchestra.run.smithi023.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-12-05T19:03:07.892 INFO:teuthology.orchestra.run.smithi023.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-12-05T19:03:07.896 INFO:teuthology.orchestra.run.smithi023.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-12-05T19:03:07.898 INFO:teuthology.orchestra.run.smithi023.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-12-05T19:03:07.898 INFO:teuthology.orchestra.run.smithi023.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-12-05T19:03:07.932 INFO:teuthology.orchestra.run.smithi185.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-12-05T19:03:07.938 INFO:teuthology.orchestra.run.smithi185.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-12-05T19:03:07.945 INFO:teuthology.orchestra.run.smithi185.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2020-12-05T19:03:07.948 INFO:teuthology.orchestra.run.smithi185.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-12-05T19:03:07.948 INFO:teuthology.orchestra.run.smithi185.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-12-05T19:03:07.962 INFO:teuthology.orchestra.run.smithi023.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-12-05T19:03:07.969 INFO:teuthology.orchestra.run.smithi023.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-12-05T19:03:08.013 INFO:teuthology.orchestra.run.smithi185.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-12-05T19:03:08.013 INFO:teuthology.orchestra.run.smithi185.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-12-05T19:03:08.015 INFO:teuthology.orchestra.run.smithi185.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-12-05T19:03:08.023 INFO:teuthology.orchestra.run.smithi185.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-12-05T19:03:08.024 INFO:teuthology.orchestra.run.smithi185.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-12-05T19:03:08.041 INFO:teuthology.orchestra.run.smithi023.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2020-12-05T19:03:08.047 INFO:teuthology.orchestra.run.smithi023.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-12-05T19:03:08.048 INFO:teuthology.orchestra.run.smithi023.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-12-05T19:03:08.049 INFO:teuthology.orchestra.run.smithi023.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-12-05T19:03:08.049 INFO:teuthology.orchestra.run.smithi023.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-12-05T19:03:08.051 INFO:teuthology.orchestra.run.smithi023.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-12-05T19:03:08.054 INFO:teuthology.orchestra.run.smithi023.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-12-05T19:03:08.054 INFO:teuthology.orchestra.run.smithi023.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-12-05T19:03:08.093 INFO:teuthology.orchestra.run.smithi185.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-12-05T19:03:08.094 INFO:teuthology.orchestra.run.smithi185.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-12-05T19:03:08.096 INFO:teuthology.orchestra.run.smithi185.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-12-05T19:03:08.108 INFO:teuthology.orchestra.run.smithi185.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.6-607-g3b1861ac-1bionic [5,170 kB] 2020-12-05T19:03:08.116 INFO:teuthology.orchestra.run.smithi185.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-12-05T19:03:08.116 INFO:teuthology.orchestra.run.smithi185.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-12-05T19:03:08.119 INFO:teuthology.orchestra.run.smithi023.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-12-05T19:03:08.119 INFO:teuthology.orchestra.run.smithi023.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-12-05T19:03:08.174 INFO:teuthology.orchestra.run.smithi185.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-12-05T19:03:08.197 INFO:teuthology.orchestra.run.smithi023.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-12-05T19:03:08.220 INFO:teuthology.orchestra.run.smithi023.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-12-05T19:03:08.220 INFO:teuthology.orchestra.run.smithi023.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-12-05T19:03:08.221 INFO:teuthology.orchestra.run.smithi023.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-12-05T19:03:08.274 INFO:teuthology.orchestra.run.smithi185.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-12-05T19:03:08.274 INFO:teuthology.orchestra.run.smithi185.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-12-05T19:03:08.275 INFO:teuthology.orchestra.run.smithi185.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-12-05T19:03:08.279 INFO:teuthology.orchestra.run.smithi185.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-12-05T19:03:08.280 INFO:teuthology.orchestra.run.smithi185.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-12-05T19:03:08.280 INFO:teuthology.orchestra.run.smithi185.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-12-05T19:03:08.281 INFO:teuthology.orchestra.run.smithi185.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-12-05T19:03:08.282 INFO:teuthology.orchestra.run.smithi185.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-12-05T19:03:08.283 INFO:teuthology.orchestra.run.smithi185.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-12-05T19:03:08.356 INFO:teuthology.orchestra.run.smithi023.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-12-05T19:03:08.357 INFO:teuthology.orchestra.run.smithi023.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-12-05T19:03:08.358 INFO:teuthology.orchestra.run.smithi023.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-12-05T19:03:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-12-05T19:03:08.366 INFO:teuthology.orchestra.run.smithi023.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-12-05T19:03:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-12-05T19:03:08.368 INFO:teuthology.orchestra.run.smithi023.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-12-05T19:03:08.370 INFO:teuthology.orchestra.run.smithi023.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-12-05T19:03:08.371 INFO:teuthology.orchestra.run.smithi023.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-12-05T19:03:08.415 INFO:teuthology.orchestra.run.smithi185.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.6-607-g3b1861ac-1bionic [148 kB] 2020-12-05T19:03:08.419 INFO:teuthology.orchestra.run.smithi185.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.6-607-g3b1861ac-1bionic [1,282 kB] 2020-12-05T19:03:08.491 INFO:teuthology.orchestra.run.smithi185.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.6-607-g3b1861ac-1bionic [5,942 kB] 2020-12-05T19:03:08.497 INFO:teuthology.orchestra.run.smithi023.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.6-607-g3b1861ac-1bionic [5,170 kB] 2020-12-05T19:03:08.854 INFO:teuthology.orchestra.run.smithi023.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.6-607-g3b1861ac-1bionic [148 kB] 2020-12-05T19:03:08.860 INFO:teuthology.orchestra.run.smithi023.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.6-607-g3b1861ac-1bionic [1,282 kB] 2020-12-05T19:03:08.863 INFO:teuthology.orchestra.run.smithi185.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.6-607-g3b1861ac-1bionic [22.7 MB] 2020-12-05T19:03:08.961 INFO:teuthology.orchestra.run.smithi023.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.6-607-g3b1861ac-1bionic [5,942 kB] 2020-12-05T19:03:09.391 INFO:teuthology.orchestra.run.smithi023.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.6-607-g3b1861ac-1bionic [22.7 MB] 2020-12-05T19:03:10.227 INFO:teuthology.orchestra.run.smithi185.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.6-607-g3b1861ac-1bionic [3,892 B] 2020-12-05T19:03:10.227 INFO:teuthology.orchestra.run.smithi185.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.6-607-g3b1861ac-1bionic [515 kB] 2020-12-05T19:03:10.256 INFO:teuthology.orchestra.run.smithi185.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.6-607-g3b1861ac-1bionic [1,850 kB] 2020-12-05T19:03:10.368 INFO:teuthology.orchestra.run.smithi185.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.6-607-g3b1861ac-1bionic [51.4 kB] 2020-12-05T19:03:10.370 INFO:teuthology.orchestra.run.smithi185.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.6-607-g3b1861ac-1bionic [55.3 kB] 2020-12-05T19:03:10.372 INFO:teuthology.orchestra.run.smithi185.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.6-607-g3b1861ac-1bionic [3,177 kB] 2020-12-05T19:03:10.566 INFO:teuthology.orchestra.run.smithi185.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.6-607-g3b1861ac-1bionic [34.3 kB] 2020-12-05T19:03:10.569 INFO:teuthology.orchestra.run.smithi185.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.6-607-g3b1861ac-1bionic [7,889 kB] 2020-12-05T19:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.6-607-g3b1861ac-1bionic [3,892 B] 2020-12-05T19:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.6-607-g3b1861ac-1bionic [515 kB] 2020-12-05T19:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.6-607-g3b1861ac-1bionic [1,850 kB] 2020-12-05T19:03:11.055 INFO:teuthology.orchestra.run.smithi185.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.6-607-g3b1861ac-1bionic [4,012 B] 2020-12-05T19:03:11.056 INFO:teuthology.orchestra.run.smithi185.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.6-607-g3b1861ac-1bionic [22.4 kB] 2020-12-05T19:03:11.057 INFO:teuthology.orchestra.run.smithi185.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.6-607-g3b1861ac-1bionic [43.9 MB] 2020-12-05T19:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.6-607-g3b1861ac-1bionic [51.4 kB] 2020-12-05T19:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.6-607-g3b1861ac-1bionic [55.3 kB] 2020-12-05T19:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.6-607-g3b1861ac-1bionic [3,177 kB] 2020-12-05T19:03:11.430 INFO:teuthology.orchestra.run.smithi023.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.6-607-g3b1861ac-1bionic [34.3 kB] 2020-12-05T19:03:11.433 INFO:teuthology.orchestra.run.smithi023.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.6-607-g3b1861ac-1bionic [7,889 kB] 2020-12-05T19:03:12.002 INFO:teuthology.orchestra.run.smithi023.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.6-607-g3b1861ac-1bionic [4,012 B] 2020-12-05T19:03:12.002 INFO:teuthology.orchestra.run.smithi023.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.6-607-g3b1861ac-1bionic [22.4 kB] 2020-12-05T19:03:12.003 INFO:teuthology.orchestra.run.smithi023.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.6-607-g3b1861ac-1bionic [43.9 MB] 2020-12-05T19:03:13.953 INFO:teuthology.orchestra.run.smithi185.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.6-607-g3b1861ac-1bionic [17.9 kB] 2020-12-05T19:03:13.953 INFO:teuthology.orchestra.run.smithi185.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.6-607-g3b1861ac-1bionic [8,845 kB] 2020-12-05T19:03:14.570 INFO:teuthology.orchestra.run.smithi185.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.6-607-g3b1861ac-1bionic [62.5 kB] 2020-12-05T19:03:14.661 INFO:teuthology.orchestra.run.smithi023.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.6-607-g3b1861ac-1bionic [17.9 kB] 2020-12-05T19:03:14.662 INFO:teuthology.orchestra.run.smithi023.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.6-607-g3b1861ac-1bionic [8,845 kB] 2020-12-05T19:03:14.948 INFO:teuthology.orchestra.run.smithi023.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/octopus/3b1861acb8137d863184795ccadbd5d0cb026674/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.6-607-g3b1861ac-1bionic [62.5 kB] 2020-12-05T19:03:15.277 INFO:teuthology.orchestra.run.smithi185.stdout:Fetched 156 MB in 8s (18.5 MB/s) 2020-12-05T19:03:15.421 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-uritemplate. 2020-12-05T19:03:15.469 INFO:teuthology.orchestra.run.smithi185.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137609 files and directories currently installed.) 2020-12-05T19:03:15.473 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-12-05T19:03:15.520 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-12-05T19:03:15.678 INFO:teuthology.orchestra.run.smithi023.stdout:Fetched 156 MB in 9s (17.9 MB/s) 2020-12-05T19:03:15.773 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-12-05T19:03:15.786 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-12-05T19:03:15.790 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-uritemplate. 2020-12-05T19:03:15.837 INFO:teuthology.orchestra.run.smithi023.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137609 files and directories currently installed.) 2020-12-05T19:03:15.841 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-12-05T19:03:15.848 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-12-05T19:03:15.872 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-12-05T19:03:16.157 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../002-librbd1_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:16.241 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-12-05T19:03:16.256 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-12-05T19:03:16.269 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking librbd1 (15.2.6-607-g3b1861ac-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-12-05T19:03:16.325 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-12-05T19:03:16.675 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../002-librbd1_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:16.772 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../003-librados2_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:16.804 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking librbd1 (15.2.6-607-g3b1861ac-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-12-05T19:03:16.899 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking librados2 (15.2.6-607-g3b1861ac-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-12-05T19:03:17.307 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../003-librados2_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:17.408 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking librados2 (15.2.6-607-g3b1861ac-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-12-05T19:03:17.512 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package libcephfs2. 2020-12-05T19:03:17.527 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../004-libcephfs2_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:17.562 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking libcephfs2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:17.814 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-rados. 2020-12-05T19:03:17.829 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../005-python3-rados_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:17.873 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-rados (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:18.054 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libcephfs2. 2020-12-05T19:03:18.069 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../004-libcephfs2_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:18.121 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libcephfs2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:18.209 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-12-05T19:03:18.222 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:18.259 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-ceph-argparse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:18.390 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-rados. 2020-12-05T19:03:18.402 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../005-python3-rados_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:18.439 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-rados (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:18.495 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-cephfs. 2020-12-05T19:03:18.510 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../007-python3-cephfs_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:18.545 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-cephfs (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:18.717 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-12-05T19:03:18.730 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:18.760 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-ceph-argparse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:18.815 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-ceph-common. 2020-12-05T19:03:18.827 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:18.865 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-ceph-common (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:19.003 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-cephfs. 2020-12-05T19:03:19.019 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../007-python3-cephfs_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:19.061 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-cephfs (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:19.168 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-prettytable. 2020-12-05T19:03:19.183 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-12-05T19:03:19.226 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-12-05T19:03:19.389 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-ceph-common. 2020-12-05T19:03:19.402 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:19.439 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-ceph-common (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:19.471 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-rbd. 2020-12-05T19:03:19.485 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../010-python3-rbd_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:19.521 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-rbd (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:19.725 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-prettytable. 2020-12-05T19:03:19.741 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-12-05T19:03:19.775 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-12-05T19:03:19.799 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-12-05T19:03:19.815 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-12-05T19:03:19.849 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-12-05T19:03:20.003 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-rbd. 2020-12-05T19:03:20.017 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../010-python3-rbd_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:20.053 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-rbd (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:20.170 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-12-05T19:03:20.184 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-12-05T19:03:20.228 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-12-05T19:03:20.406 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-12-05T19:03:20.420 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-12-05T19:03:20.464 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-12-05T19:03:20.514 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package librgw2. 2020-12-05T19:03:20.528 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../013-librgw2_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:20.564 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking librgw2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:20.775 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-12-05T19:03:20.791 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-12-05T19:03:20.834 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-12-05T19:03:21.025 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-rgw. 2020-12-05T19:03:21.038 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../014-python3-rgw_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:21.075 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-rgw (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:21.120 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package librgw2. 2020-12-05T19:03:21.133 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../013-librgw2_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:21.161 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking librgw2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:21.403 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-12-05T19:03:21.417 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-12-05T19:03:21.461 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-12-05T19:03:21.647 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-rgw. 2020-12-05T19:03:21.663 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../014-python3-rgw_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:21.705 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-rgw (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:21.714 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-12-05T19:03:21.730 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-12-05T19:03:21.763 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-12-05T19:03:22.000 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-12-05T19:03:22.015 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-12-05T19:03:22.058 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-12-05T19:03:22.058 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package liboath0. 2020-12-05T19:03:22.074 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-12-05T19:03:22.108 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-12-05T19:03:22.336 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-12-05T19:03:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-12-05T19:03:22.385 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-12-05T19:03:22.713 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package liboath0. 2020-12-05T19:03:22.733 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-12-05T19:03:22.862 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package libradosstriper1. 2020-12-05T19:03:22.874 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../018-libradosstriper1_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:22.912 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking libradosstriper1 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:22.963 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-12-05T19:03:23.283 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libradosstriper1. 2020-12-05T19:03:23.290 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-common. 2020-12-05T19:03:23.297 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../018-libradosstriper1_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:23.303 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../019-ceph-common_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:23.332 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-common (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:23.341 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libradosstriper1 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:23.736 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-common. 2020-12-05T19:03:23.752 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../019-ceph-common_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:23.786 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-common (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:25.138 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-base. 2020-12-05T19:03:25.151 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../020-ceph-base_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:25.205 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-base (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:25.533 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-base. 2020-12-05T19:03:25.546 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../020-ceph-base_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:25.600 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-base (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:26.093 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-dateutil. 2020-12-05T19:03:26.109 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-12-05T19:03:26.143 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-12-05T19:03:26.421 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-asn1crypto. 2020-12-05T19:03:26.435 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-12-05T19:03:26.479 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-12-05T19:03:26.529 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-dateutil. 2020-12-05T19:03:26.542 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-12-05T19:03:26.587 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-12-05T19:03:26.749 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-cffi-backend. 2020-12-05T19:03:26.763 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-12-05T19:03:26.807 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-12-05T19:03:26.890 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-asn1crypto. 2020-12-05T19:03:26.905 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-12-05T19:03:26.948 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-12-05T19:03:27.110 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-cryptography. 2020-12-05T19:03:27.123 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-12-05T19:03:27.160 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-12-05T19:03:27.226 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-cffi-backend. 2020-12-05T19:03:27.241 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-12-05T19:03:27.284 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-12-05T19:03:27.554 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-cryptography. 2020-12-05T19:03:27.556 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-openssl. 2020-12-05T19:03:27.569 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-12-05T19:03:27.570 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-12-05T19:03:27.603 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-12-05T19:03:27.622 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-12-05T19:03:27.892 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-12-05T19:03:27.905 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:27.942 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mgr-modules-core (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:28.074 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-openssl. 2020-12-05T19:03:28.090 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-12-05T19:03:28.124 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-12-05T19:03:28.280 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-bcrypt. 2020-12-05T19:03:28.294 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-12-05T19:03:28.338 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-12-05T19:03:28.376 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-12-05T19:03:28.390 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:28.435 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-modules-core (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:28.650 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-cherrypy3. 2020-12-05T19:03:28.665 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-12-05T19:03:28.708 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-12-05T19:03:28.806 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-bcrypt. 2020-12-05T19:03:28.819 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-12-05T19:03:28.856 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-12-05T19:03:29.062 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-jwt. 2020-12-05T19:03:29.077 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-12-05T19:03:29.120 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-12-05T19:03:29.158 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-cherrypy3. 2020-12-05T19:03:29.172 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-12-05T19:03:29.208 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-12-05T19:03:29.441 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-logutils. 2020-12-05T19:03:29.454 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-12-05T19:03:29.498 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-12-05T19:03:29.570 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-jwt. 2020-12-05T19:03:29.586 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-12-05T19:03:29.628 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-12-05T19:03:29.810 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-markupsafe. 2020-12-05T19:03:29.823 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-12-05T19:03:29.860 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-12-05T19:03:29.931 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-logutils. 2020-12-05T19:03:29.947 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-12-05T19:03:29.998 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-12-05T19:03:30.155 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-mako. 2020-12-05T19:03:30.169 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-12-05T19:03:30.213 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-12-05T19:03:30.309 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-markupsafe. 2020-12-05T19:03:30.326 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-12-05T19:03:30.367 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-12-05T19:03:30.508 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-simplegeneric. 2020-12-05T19:03:30.521 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-12-05T19:03:30.566 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-12-05T19:03:30.661 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-mako. 2020-12-05T19:03:30.676 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-12-05T19:03:30.720 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-12-05T19:03:30.869 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-singledispatch. 2020-12-05T19:03:30.882 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-12-05T19:03:30.919 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-12-05T19:03:31.023 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-simplegeneric. 2020-12-05T19:03:31.037 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-12-05T19:03:31.081 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-12-05T19:03:31.231 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-webob. 2020-12-05T19:03:31.245 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-12-05T19:03:31.289 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-12-05T19:03:31.384 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-singledispatch. 2020-12-05T19:03:31.398 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-12-05T19:03:31.442 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-12-05T19:03:31.592 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-bs4. 2020-12-05T19:03:31.606 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-12-05T19:03:31.650 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-12-05T19:03:31.735 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-webob. 2020-12-05T19:03:31.751 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-12-05T19:03:31.801 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-12-05T19:03:31.970 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-waitress. 2020-12-05T19:03:31.985 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-12-05T19:03:32.029 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-12-05T19:03:32.162 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-bs4. 2020-12-05T19:03:32.177 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-12-05T19:03:32.229 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-12-05T19:03:32.332 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-tempita. 2020-12-05T19:03:32.347 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-12-05T19:03:32.390 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-12-05T19:03:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-waitress. 2020-12-05T19:03:32.598 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-12-05T19:03:32.640 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-12-05T19:03:32.701 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-paste. 2020-12-05T19:03:32.714 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-12-05T19:03:32.760 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-12-05T19:03:32.977 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-tempita. 2020-12-05T19:03:32.992 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-12-05T19:03:33.035 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-12-05T19:03:33.122 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-12-05T19:03:33.137 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-12-05T19:03:33.180 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-12-05T19:03:33.330 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-paste. 2020-12-05T19:03:33.345 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-12-05T19:03:33.388 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-12-05T19:03:33.479 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pastedeploy. 2020-12-05T19:03:33.493 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-12-05T19:03:33.537 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-12-05T19:03:33.699 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-12-05T19:03:33.713 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-12-05T19:03:33.758 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-12-05T19:03:33.857 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-webtest. 2020-12-05T19:03:33.870 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-12-05T19:03:33.915 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-12-05T19:03:34.038 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pastedeploy. 2020-12-05T19:03:34.054 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-12-05T19:03:34.096 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-12-05T19:03:34.252 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pecan. 2020-12-05T19:03:34.266 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-12-05T19:03:34.310 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-12-05T19:03:34.407 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-webtest. 2020-12-05T19:03:34.422 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-12-05T19:03:34.474 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-12-05T19:03:34.649 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-werkzeug. 2020-12-05T19:03:34.662 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2020-12-05T19:03:34.698 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2020-12-05T19:03:34.835 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pecan. 2020-12-05T19:03:34.851 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-12-05T19:03:34.893 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-12-05T19:03:35.060 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mgr. 2020-12-05T19:03:35.074 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../045-ceph-mgr_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:35.118 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mgr (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:35.215 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-werkzeug. 2020-12-05T19:03:35.231 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2020-12-05T19:03:35.273 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2020-12-05T19:03:35.555 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mon. 2020-12-05T19:03:35.571 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../046-ceph-mon_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:35.614 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mon (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:35.659 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr. 2020-12-05T19:03:35.675 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../045-ceph-mgr_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:35.717 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:36.171 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mon. 2020-12-05T19:03:36.185 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../046-ceph-mon_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:36.229 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mon (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:36.534 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-osd. 2020-12-05T19:03:36.547 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../047-ceph-osd_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:36.617 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-osd (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:37.124 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-osd. 2020-12-05T19:03:37.140 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../047-ceph-osd_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:37.182 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-osd (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:39.123 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph. 2020-12-05T19:03:39.139 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../048-ceph_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:39.182 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:39.421 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph. 2020-12-05T19:03:39.437 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../048-ceph_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:39.509 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-fuse. 2020-12-05T19:03:39.524 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../049-ceph-fuse_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:39.568 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-fuse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-fuse. 2020-12-05T19:03:39.812 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../049-ceph-fuse_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:39.848 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-fuse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:39.947 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mds. 2020-12-05T19:03:39.961 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../050-ceph-mds_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:40.005 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mds (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:40.252 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mds. 2020-12-05T19:03:40.266 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../050-ceph-mds_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:40.310 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mds (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:40.533 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package cephadm. 2020-12-05T19:03:40.548 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../051-cephadm_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:40.591 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking cephadm (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:40.788 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package cephadm. 2020-12-05T19:03:40.801 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../051-cephadm_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:40.846 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking cephadm (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:40.886 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-jinja2. 2020-12-05T19:03:40.902 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-12-05T19:03:40.944 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-12-05T19:03:41.166 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-jinja2. 2020-12-05T19:03:41.181 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-12-05T19:03:41.224 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-12-05T19:03:41.273 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-12-05T19:03:41.286 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:41.331 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mgr-cephadm (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:41.594 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-12-05T19:03:41.610 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:41.651 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-repoze.lru. 2020-12-05T19:03:41.667 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-12-05T19:03:41.668 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-cephadm (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:41.709 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-12-05T19:03:42.030 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-repoze.lru. 2020-12-05T19:03:42.037 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-routes. 2020-12-05T19:03:42.043 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-12-05T19:03:42.050 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-12-05T19:03:42.096 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-12-05T19:03:42.105 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-12-05T19:03:42.382 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-12-05T19:03:42.398 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:42.418 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-routes. 2020-12-05T19:03:42.434 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-12-05T19:03:42.440 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mgr-dashboard (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:42.476 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-12-05T19:03:42.762 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-12-05T19:03:42.779 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:42.820 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-dashboard (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:43.299 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-12-05T19:03:43.316 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:43.357 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:43.694 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-numpy. 2020-12-05T19:03:43.707 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-12-05T19:03:43.719 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-12-05T19:03:43.736 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:43.761 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-12-05T19:03:43.786 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:44.114 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-numpy. 2020-12-05T19:03:44.127 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-12-05T19:03:44.172 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-12-05T19:03:44.342 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-decorator. 2020-12-05T19:03:44.355 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2020-12-05T19:03:44.392 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-12-05T19:03:44.703 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-scipy. 2020-12-05T19:03:44.719 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-12-05T19:03:44.762 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-12-05T19:03:44.811 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-decorator. 2020-12-05T19:03:44.825 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2020-12-05T19:03:44.869 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-12-05T19:03:45.206 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-scipy. 2020-12-05T19:03:45.219 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-12-05T19:03:45.264 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-12-05T19:03:46.073 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-12-05T19:03:46.090 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-12-05T19:03:46.131 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-12-05T19:03:46.509 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-12-05T19:03:46.523 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-12-05T19:03:46.567 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-12-05T19:03:46.598 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-joblib. 2020-12-05T19:03:46.614 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2020-12-05T19:03:46.656 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-joblib (0.11-1) ... 2020-12-05T19:03:47.009 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-sklearn. 2020-12-05T19:03:47.026 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2020-12-05T19:03:47.037 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-joblib. 2020-12-05T19:03:47.051 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2020-12-05T19:03:47.068 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-12-05T19:03:47.095 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-joblib (0.11-1) ... 2020-12-05T19:03:47.440 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-sklearn. 2020-12-05T19:03:47.454 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2020-12-05T19:03:47.507 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-12-05T19:03:47.674 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-12-05T19:03:47.689 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:47.749 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:48.072 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-12-05T19:03:48.088 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:48.138 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:48.861 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-monotonic. 2020-12-05T19:03:48.875 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2020-12-05T19:03:48.919 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-12-05T19:03:49.214 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-fasteners. 2020-12-05T19:03:49.227 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2020-12-05T19:03:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-monotonic. 2020-12-05T19:03:49.255 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2020-12-05T19:03:49.264 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-12-05T19:03:49.300 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-12-05T19:03:49.566 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-httplib2. 2020-12-05T19:03:49.582 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-12-05T19:03:49.663 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-fasteners. 2020-12-05T19:03:49.664 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2020-12-05T19:03:49.664 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-12-05T19:03:49.665 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-12-05T19:03:49.926 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pyasn1. 2020-12-05T19:03:49.942 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2020-12-05T19:03:49.970 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-httplib2. 2020-12-05T19:03:49.984 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-12-05T19:03:49.985 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-12-05T19:03:50.029 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-12-05T19:03:50.331 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pyasn1. 2020-12-05T19:03:50.332 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-12-05T19:03:50.345 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-12-05T19:03:50.346 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2020-12-05T19:03:50.389 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-12-05T19:03:50.398 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-12-05T19:03:50.745 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-12-05T19:03:50.751 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-rsa. 2020-12-05T19:03:50.759 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-12-05T19:03:50.765 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2020-12-05T19:03:50.803 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-12-05T19:03:50.809 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-12-05T19:03:51.148 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-rsa. 2020-12-05T19:03:51.154 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-12-05T19:03:51.162 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2020-12-05T19:03:51.169 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-12-05T19:03:51.206 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-12-05T19:03:51.212 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-12-05T19:03:51.555 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-12-05T19:03:51.569 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-12-05T19:03:51.613 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-12-05T19:03:51.646 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-oauth2client. 2020-12-05T19:03:51.663 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2020-12-05T19:03:51.704 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-12-05T19:03:52.041 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-setuptools. 2020-12-05T19:03:52.055 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2020-12-05T19:03:52.084 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-oauth2client. 2020-12-05T19:03:52.099 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-12-05T19:03:52.100 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2020-12-05T19:03:52.150 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-12-05T19:03:52.470 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-setuptools. 2020-12-05T19:03:52.486 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-websocket. 2020-12-05T19:03:52.487 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2020-12-05T19:03:52.501 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-12-05T19:03:52.528 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-12-05T19:03:52.544 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-12-05T19:03:52.858 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-kubernetes. 2020-12-05T19:03:52.873 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-12-05T19:03:52.916 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-12-05T19:03:52.957 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-websocket. 2020-12-05T19:03:52.973 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-12-05T19:03:53.023 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-12-05T19:03:53.343 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-kubernetes. 2020-12-05T19:03:53.358 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-12-05T19:03:53.504 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-12-05T19:03:53.720 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-12-05T19:03:53.736 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:53.778 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mgr-k8sevents (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:54.098 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-json-pointer. 2020-12-05T19:03:54.111 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2020-12-05T19:03:54.164 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-12-05T19:03:54.264 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-12-05T19:03:54.278 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:54.322 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-k8sevents (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:54.543 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-jsonpatch. 2020-12-05T19:03:54.557 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-12-05T19:03:54.604 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-12-05T19:03:54.683 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-json-pointer. 2020-12-05T19:03:54.698 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2020-12-05T19:03:54.749 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-12-05T19:03:54.913 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-12-05T19:03:54.926 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:54.963 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-mgr-rook (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:55.119 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-jsonpatch. 2020-12-05T19:03:55.134 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-12-05T19:03:55.180 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-12-05T19:03:55.291 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package libonig4:amd64. 2020-12-05T19:03:55.305 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2020-12-05T19:03:55.443 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-12-05T19:03:55.481 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-12-05T19:03:55.497 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.6-607-g3b1861ac-1bionic_all.deb ... 2020-12-05T19:03:55.539 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-mgr-rook (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:55.805 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package libjq1:amd64. 2020-12-05T19:03:55.818 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2020-12-05T19:03:55.863 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-12-05T19:03:55.876 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libonig4:amd64. 2020-12-05T19:03:55.890 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2020-12-05T19:03:56.020 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-12-05T19:03:56.124 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package jq. 2020-12-05T19:03:56.138 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2020-12-05T19:03:56.182 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-12-05T19:03:56.381 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libjq1:amd64. 2020-12-05T19:03:56.398 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2020-12-05T19:03:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-12-05T19:03:56.468 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package socat. 2020-12-05T19:03:56.485 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-12-05T19:03:56.527 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-12-05T19:03:56.700 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package jq. 2020-12-05T19:03:56.714 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2020-12-05T19:03:56.758 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-12-05T19:03:56.887 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-12-05T19:03:56.902 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-12-05T19:03:56.938 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-12-05T19:03:57.036 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package socat. 2020-12-05T19:03:57.052 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-12-05T19:03:57.095 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-12-05T19:03:57.183 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package xmlstarlet. 2020-12-05T19:03:57.196 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2020-12-05T19:03:57.224 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-12-05T19:03:57.481 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-12-05T19:03:57.495 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-12-05T19:03:57.539 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-12-05T19:03:57.545 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package ceph-test. 2020-12-05T19:03:57.562 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../086-ceph-test_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:57.595 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking ceph-test (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:03:57.817 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package xmlstarlet. 2020-12-05T19:03:57.833 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2020-12-05T19:03:57.883 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-12-05T19:03:58.229 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package ceph-test. 2020-12-05T19:03:58.245 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../086-ceph-test_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:03:58.287 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking ceph-test (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:00.899 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package libcephfs-dev. 2020-12-05T19:04:00.912 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:04:00.957 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking libcephfs-dev (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:01.251 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package nvme-cli. 2020-12-05T19:04:01.266 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2020-12-05T19:04:01.309 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2020-12-05T19:04:01.591 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package libcephfs-dev. 2020-12-05T19:04:01.605 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:04:01.623 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python-attr. 2020-12-05T19:04:01.639 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2020-12-05T19:04:01.649 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking libcephfs-dev (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:01.698 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python-attr (17.4.0-2) ... 2020-12-05T19:04:01.935 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package nvme-cli. 2020-12-05T19:04:01.942 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python-funcsigs. 2020-12-05T19:04:01.950 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2020-12-05T19:04:01.956 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2020-12-05T19:04:01.992 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-12-05T19:04:01.993 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2020-12-05T19:04:02.303 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python-pluggy. 2020-12-05T19:04:02.318 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2020-12-05T19:04:02.323 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python-attr. 2020-12-05T19:04:02.340 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2020-12-05T19:04:02.370 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-12-05T19:04:02.373 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python-attr (17.4.0-2) ... 2020-12-05T19:04:02.668 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python-funcsigs. 2020-12-05T19:04:02.672 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python-py. 2020-12-05T19:04:02.682 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2020-12-05T19:04:02.686 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2020-12-05T19:04:02.726 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-12-05T19:04:02.731 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python-py (1.5.2-1) ... 2020-12-05T19:04:03.029 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python-pluggy. 2020-12-05T19:04:03.043 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2020-12-05T19:04:03.059 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python-pytest. 2020-12-05T19:04:03.074 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2020-12-05T19:04:03.087 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-12-05T19:04:03.117 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-12-05T19:04:03.365 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python-py. 2020-12-05T19:04:03.381 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2020-12-05T19:04:03.414 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python-py (1.5.2-1) ... 2020-12-05T19:04:03.437 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-attr. 2020-12-05T19:04:03.454 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2020-12-05T19:04:03.495 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-12-05T19:04:03.693 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python-pytest. 2020-12-05T19:04:03.707 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2020-12-05T19:04:03.743 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-12-05T19:04:03.782 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-cycler. 2020-12-05T19:04:03.798 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2020-12-05T19:04:03.840 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-12-05T19:04:04.013 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-attr. 2020-12-05T19:04:04.028 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2020-12-05T19:04:04.062 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-12-05T19:04:04.143 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-webencodings. 2020-12-05T19:04:04.158 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2020-12-05T19:04:04.201 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-12-05T19:04:04.290 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-cycler. 2020-12-05T19:04:04.304 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2020-12-05T19:04:04.340 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-12-05T19:04:04.504 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-html5lib. 2020-12-05T19:04:04.518 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2020-12-05T19:04:04.562 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-12-05T19:04:04.626 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-webencodings. 2020-12-05T19:04:04.643 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2020-12-05T19:04:04.684 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-12-05T19:04:04.899 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-12-05T19:04:04.913 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-12-05T19:04:04.957 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-12-05T19:04:04.970 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-html5lib. 2020-12-05T19:04:04.987 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2020-12-05T19:04:05.029 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-12-05T19:04:05.382 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-12-05T19:04:05.397 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-12-05T19:04:05.440 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-12-05T19:04:05.469 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pyparsing. 2020-12-05T19:04:05.485 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-12-05T19:04:05.527 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-12-05T19:04:05.847 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-tz. 2020-12-05T19:04:05.852 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pyparsing. 2020-12-05T19:04:05.861 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2020-12-05T19:04:05.868 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-12-05T19:04:05.905 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-tz (2018.3-2) ... 2020-12-05T19:04:05.910 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-12-05T19:04:06.233 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-matplotlib. 2020-12-05T19:04:06.247 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-12-05T19:04:06.254 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-tz. 2020-12-05T19:04:06.270 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2020-12-05T19:04:06.291 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-12-05T19:04:06.313 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-tz (2018.3-2) ... 2020-12-05T19:04:06.616 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-matplotlib. 2020-12-05T19:04:06.630 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-12-05T19:04:06.674 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-12-05T19:04:07.039 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-olefile. 2020-12-05T19:04:07.054 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2020-12-05T19:04:07.097 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-12-05T19:04:07.355 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-olefile. 2020-12-05T19:04:07.372 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2020-12-05T19:04:07.413 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-12-05T19:04:07.417 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pastescript. 2020-12-05T19:04:07.433 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2020-12-05T19:04:07.475 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-12-05T19:04:07.733 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pastescript. 2020-12-05T19:04:07.747 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2020-12-05T19:04:07.791 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-12-05T19:04:07.829 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pil:amd64. 2020-12-05T19:04:07.845 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-12-05T19:04:07.887 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-12-05T19:04:08.153 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pil:amd64. 2020-12-05T19:04:08.167 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-12-05T19:04:08.211 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-12-05T19:04:08.241 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pluggy. 2020-12-05T19:04:08.258 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2020-12-05T19:04:08.299 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-12-05T19:04:08.581 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pluggy. 2020-12-05T19:04:08.585 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-py. 2020-12-05T19:04:08.595 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2020-12-05T19:04:08.599 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2020-12-05T19:04:08.640 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-12-05T19:04:08.643 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-py (1.5.2-1) ... 2020-12-05T19:04:08.959 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-py. 2020-12-05T19:04:08.974 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2020-12-05T19:04:08.980 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pyinotify. 2020-12-05T19:04:08.994 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2020-12-05T19:04:09.017 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-py (1.5.2-1) ... 2020-12-05T19:04:09.030 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-12-05T19:04:09.299 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-pytest. 2020-12-05T19:04:09.316 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2020-12-05T19:04:09.357 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-12-05T19:04:09.362 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pyinotify. 2020-12-05T19:04:09.376 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2020-12-05T19:04:09.420 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-12-05T19:04:09.636 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-simplejson. 2020-12-05T19:04:09.653 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2020-12-05T19:04:09.686 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-12-05T19:04:09.731 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-pytest. 2020-12-05T19:04:09.746 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2020-12-05T19:04:09.790 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-12-05T19:04:09.964 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package python3-tk:amd64. 2020-12-05T19:04:09.980 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-12-05T19:04:10.022 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-12-05T19:04:10.118 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-simplejson. 2020-12-05T19:04:10.136 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2020-12-05T19:04:10.177 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-12-05T19:04:10.484 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package radosgw. 2020-12-05T19:04:10.498 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../111-radosgw_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:04:10.504 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package python3-tk:amd64. 2020-12-05T19:04:10.519 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-12-05T19:04:10.542 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking radosgw (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:10.563 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-12-05T19:04:11.033 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package radosgw. 2020-12-05T19:04:11.047 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../111-radosgw_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:04:11.099 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking radosgw (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:11.454 INFO:teuthology.orchestra.run.smithi185.stdout:Selecting previously unselected package rbd-fuse. 2020-12-05T19:04:11.469 INFO:teuthology.orchestra.run.smithi185.stdout:Preparing to unpack .../112-rbd-fuse_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:04:11.504 INFO:teuthology.orchestra.run.smithi185.stdout:Unpacking rbd-fuse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:11.788 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-12-05T19:04:12.078 INFO:teuthology.orchestra.run.smithi023.stdout:Selecting previously unselected package rbd-fuse. 2020-12-05T19:04:12.092 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2020-12-05T19:04:12.093 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing to unpack .../112-rbd-fuse_15.2.6-607-g3b1861ac-1bionic_amd64.deb ... 2020-12-05T19:04:12.136 INFO:teuthology.orchestra.run.smithi023.stdout:Unpacking rbd-fuse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:12.460 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-12-05T19:04:12.463 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-12-05T19:04:12.578 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-12-05T19:04:12.799 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2020-12-05T19:04:13.167 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-12-05T19:04:13.172 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-12-05T19:04:13.293 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-12-05T19:04:13.540 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-12-05T19:04:13.816 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-webencodings (0.5-2) ... 2020-12-05T19:04:13.904 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-12-05T19:04:14.126 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python-attr (17.4.0-2) ... 2020-12-05T19:04:14.272 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-12-05T19:04:14.327 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-12-05T19:04:14.436 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-py (1.5.2-1) ... 2020-12-05T19:04:14.531 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-webencodings (0.5-2) ... 2020-12-05T19:04:14.755 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-12-05T19:04:14.816 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python-attr (17.4.0-2) ... 2020-12-05T19:04:14.901 INFO:teuthology.orchestra.run.smithi185.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-12-05T19:04:15.017 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-12-05T19:04:15.143 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-py (1.5.2-1) ... 2020-12-05T19:04:15.199 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-12-05T19:04:15.461 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-12-05T19:04:15.492 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-ceph-argparse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:15.656 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-12-05T19:04:15.793 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-12-05T19:04:15.922 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-12-05T19:04:16.103 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-12-05T19:04:16.204 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-12-05T19:04:16.207 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-ceph-argparse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:16.508 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-12-05T19:04:16.809 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-12-05T19:04:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-12-05T19:04:17.090 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-12-05T19:04:17.391 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-12-05T19:04:17.701 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python-py (1.5.2-1) ... 2020-12-05T19:04:17.880 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-12-05T19:04:17.935 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-12-05T19:04:18.189 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-12-05T19:04:18.278 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-12-05T19:04:18.376 INFO:teuthology.orchestra.run.smithi185.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-12-05T19:04:18.474 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python-py (1.5.2-1) ... 2020-12-05T19:04:18.605 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-12-05T19:04:18.705 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-12-05T19:04:18.716 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-12-05T19:04:18.923 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-12-05T19:04:19.159 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-12-05T19:04:19.216 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-12-05T19:04:19.241 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-12-05T19:04:19.478 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-12-05T19:04:19.551 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-12-05T19:04:19.578 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-12-05T19:04:19.763 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-12-05T19:04:19.852 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-12-05T19:04:20.039 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-12-05T19:04:20.220 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-12-05T19:04:20.348 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-12-05T19:04:20.529 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-12-05T19:04:20.649 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-12-05T19:04:20.797 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up cephadm (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:21.017 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-12-05T19:04:21.352 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-12-05T19:04:21.465 INFO:teuthology.orchestra.run.smithi185.stdout:Adding system user cephadm....done 2020-12-05T19:04:21.612 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up cephadm (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:21.629 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-12-05T19:04:21.897 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-12-05T19:04:21.997 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2020-12-05T19:04:22.115 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-12-05T19:04:22.195 INFO:teuthology.orchestra.run.smithi023.stdout:Adding system user cephadm....done 2020-12-05T19:04:22.351 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-12-05T19:04:22.433 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-12-05T19:04:22.652 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-12-05T19:04:22.709 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-12-05T19:04:22.762 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2020-12-05T19:04:22.835 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-12-05T19:04:22.879 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-12-05T19:04:23.204 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-ceph-common (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:23.214 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-12-05T19:04:23.497 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-monotonic (1.1-2) ... 2020-12-05T19:04:23.506 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-12-05T19:04:23.624 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-12-05T19:04:23.756 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-12-05T19:04:23.857 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-12-05T19:04:23.967 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-ceph-common (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:24.408 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-12-05T19:04:24.594 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-monotonic (1.1-2) ... 2020-12-05T19:04:24.776 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-12-05T19:04:24.866 INFO:teuthology.orchestra.run.smithi185.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-12-05T19:04:24.890 INFO:teuthology.orchestra.run.smithi185.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-12-05T19:04:24.903 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-12-05T19:04:25.004 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-12-05T19:04:25.120 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-12-05T19:04:25.338 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-12-05T19:04:25.413 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-12-05T19:04:25.665 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-12-05T19:04:25.689 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-12-05T19:04:25.738 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-12-05T19:04:25.762 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-12-05T19:04:25.998 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-attr (17.4.0-2) ... 2020-12-05T19:04:26.000 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-12-05T19:04:26.283 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-12-05T19:04:26.301 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-12-05T19:04:26.476 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-12-05T19:04:26.619 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-12-05T19:04:26.885 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-12-05T19:04:26.945 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-attr (17.4.0-2) ... 2020-12-05T19:04:26.975 INFO:teuthology.orchestra.run.smithi185.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-12-05T19:04:27.237 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up liboath0 (2.6.1-1) ... 2020-12-05T19:04:27.238 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-12-05T19:04:27.380 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-12-05T19:04:27.422 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-12-05T19:04:27.481 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-12-05T19:04:27.716 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-12-05T19:04:27.824 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-12-05T19:04:27.825 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-12-05T19:04:27.914 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-12-05T19:04:28.176 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up liboath0 (2.6.1-1) ... 2020-12-05T19:04:28.227 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-12-05T19:04:28.353 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-12-05T19:04:28.360 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-12-05T19:04:28.486 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-12-05T19:04:28.645 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-tz (2018.3-2) ... 2020-12-05T19:04:28.712 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-12-05T19:04:28.813 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-12-05T19:04:28.963 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-12-05T19:04:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-12-05T19:04:29.239 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-12-05T19:04:29.291 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-12-05T19:04:29.515 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-joblib (0.11-1) ... 2020-12-05T19:04:29.617 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-tz (2018.3-2) ... 2020-12-05T19:04:29.850 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-routes (2.4.1-1) ... 2020-12-05T19:04:29.935 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-12-05T19:04:30.152 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-12-05T19:04:30.228 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-12-05T19:04:30.453 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-12-05T19:04:30.529 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-joblib (0.11-1) ... 2020-12-05T19:04:30.779 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-12-05T19:04:30.863 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-routes (2.4.1-1) ... 2020-12-05T19:04:31.147 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-12-05T19:04:31.156 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-12-05T19:04:31.423 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up jq (1.5+dfsg-2) ... 2020-12-05T19:04:31.440 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-12-05T19:04:31.549 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-12-05T19:04:31.759 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-12-05T19:04:32.137 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-12-05T19:04:32.413 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up jq (1.5+dfsg-2) ... 2020-12-05T19:04:32.514 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-12-05T19:04:32.696 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-12-05T19:04:32.989 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-12-05T19:04:33.299 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-12-05T19:04:33.400 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-12-05T19:04:33.703 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-12-05T19:04:33.751 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python-pytest (3.3.2-2) ... 2020-12-05T19:04:34.003 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-12-05T19:04:34.021 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-12-05T19:04:34.129 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-12-05T19:04:34.263 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up librados2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:34.322 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-12-05T19:04:34.372 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-12-05T19:04:34.431 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-12-05T19:04:34.665 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up libcephfs2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:34.766 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up libcephfs-dev (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:34.774 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python-pytest (3.3.2-2) ... 2020-12-05T19:04:34.875 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-rados (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:34.993 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-12-05T19:04:35.025 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-12-05T19:04:35.151 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-12-05T19:04:35.277 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up librados2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:35.311 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-12-05T19:04:35.412 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-12-05T19:04:35.612 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-12-05T19:04:35.696 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libcephfs2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:35.797 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libcephfs-dev (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:35.898 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-rados (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:35.939 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-12-05T19:04:35.999 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-12-05T19:04:36.325 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-12-05T19:04:36.635 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-12-05T19:04:36.860 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-12-05T19:04:36.969 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-12-05T19:04:37.681 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-12-05T19:04:37.932 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-12-05T19:04:37.957 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up libradosstriper1 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:38.058 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-cephfs (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:38.353 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up librgw2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:38.479 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-fuse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:38.694 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-12-05T19:04:38.695 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-12-05T19:04:38.770 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-12-05T19:04:38.963 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up librbd1 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:39.046 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up libradosstriper1 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:39.064 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-12-05T19:04:39.138 INFO:teuthology.orchestra.run.smithi185.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-12-05T19:04:39.147 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-cephfs (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:39.178 INFO:teuthology.orchestra.run.smithi185.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-12-05T19:04:39.432 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up librgw2 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:39.477 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-12-05T19:04:39.541 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-fuse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:39.759 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-12-05T19:04:39.759 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-12-05T19:04:40.051 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up librbd1 (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:40.177 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-12-05T19:04:40.229 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mgr-modules-core (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:40.267 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-12-05T19:04:40.299 INFO:teuthology.orchestra.run.smithi023.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-12-05T19:04:40.355 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-12-05T19:04:40.606 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-12-05T19:04:40.640 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-rgw (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:40.741 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up rbd-fuse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:40.842 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up python3-rbd (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:40.943 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-common (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:41.241 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-modules-core (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:41.367 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-12-05T19:04:41.486 INFO:teuthology.orchestra.run.smithi185.stdout:Adding group ceph....done 2020-12-05T19:04:41.677 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-rgw (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:41.796 INFO:teuthology.orchestra.run.smithi185.stdout:Adding system user ceph....done 2020-12-05T19:04:41.802 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up rbd-fuse (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:41.929 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up python3-rbd (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:41.972 INFO:teuthology.orchestra.run.smithi185.stdout:Setting system user ceph properties....done 2020-12-05T19:04:42.030 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-common (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:42.135 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-12-05T19:04:42.362 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-12-05T19:04:42.521 INFO:teuthology.orchestra.run.smithi023.stdout:Adding group ceph....done 2020-12-05T19:04:42.670 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up radosgw (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:42.908 INFO:teuthology.orchestra.run.smithi023.stdout:Adding system user ceph....done 2020-12-05T19:04:42.931 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-12-05T19:04:42.931 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-12-05T19:04:43.101 INFO:teuthology.orchestra.run.smithi023.stdout:Setting system user ceph properties....done 2020-12-05T19:04:43.252 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-12-05T19:04:43.313 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-test (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:43.439 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-base (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:43.477 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-12-05T19:04:43.743 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-12-05T19:04:43.765 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up radosgw (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:44.005 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-12-05T19:04:44.006 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-12-05T19:04:44.007 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mgr (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:44.199 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-12-05T19:04:44.200 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-12-05T19:04:44.433 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-test (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:44.551 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mgr-diskprediction-local (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:44.559 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-base (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:44.694 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-osd (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:45.024 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-12-05T19:04:45.165 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-12-05T19:04:45.166 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-12-05T19:04:45.328 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:45.421 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mgr-cephadm (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:45.549 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-12-05T19:04:45.549 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-12-05T19:04:45.564 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:45.690 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mgr-rook (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:45.808 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mds (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:45.821 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-diskprediction-local (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:45.939 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-osd (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:46.011 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-12-05T19:04:46.012 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-12-05T19:04:46.276 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mon (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:46.369 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-12-05T19:04:46.370 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-12-05T19:04:46.486 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-12-05T19:04:46.486 INFO:teuthology.orchestra.run.smithi185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-12-05T19:04:46.650 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-cephadm (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:46.776 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:46.786 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mgr-k8sevents (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:46.918 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-rook (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:46.929 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph-mgr-dashboard (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:47.061 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mds (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:47.071 INFO:teuthology.orchestra.run.smithi185.stdout:Setting up ceph (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:47.198 INFO:teuthology.orchestra.run.smithi185.stdout:Processing triggers for systemd (237-3ubuntu10.43) ... 2020-12-05T19:04:47.277 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-12-05T19:04:47.278 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-12-05T19:04:47.382 INFO:teuthology.orchestra.run.smithi185.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-12-05T19:04:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mon (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:47.784 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-12-05T19:04:47.785 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-12-05T19:04:48.064 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-k8sevents (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:48.165 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph-mgr-dashboard (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:48.283 INFO:teuthology.orchestra.run.smithi023.stdout:Setting up ceph (15.2.6-607-g3b1861ac-1bionic) ... 2020-12-05T19:04:48.392 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for systemd (237-3ubuntu10.43) ... 2020-12-05T19:04:48.577 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-12-05T19:04:49.435 INFO:teuthology.orchestra.run.smithi185.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-12-05T19:04:49.469 INFO:teuthology.orchestra.run.smithi185.stdout:ureadahead will be reprofiled on next reboot 2020-12-05T19:04:49.519 INFO:teuthology.orchestra.run.smithi185.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-12-05T19:04:50.121 INFO:teuthology.orchestra.run.smithi185.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-12-05T19:04:50.122 DEBUG:teuthology.parallel:result is None 2020-12-05T19:04:50.673 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-12-05T19:04:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:ureadahead will be reprofiled on next reboot 2020-12-05T19:04:50.756 INFO:teuthology.orchestra.run.smithi023.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-12-05T19:04:51.197 INFO:teuthology.orchestra.run.smithi023.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-12-05T19:04:51.199 DEBUG:teuthology.parallel:result is None 2020-12-05T19:04:51.200 INFO:teuthology.orchestra.run.smithi185:> sudo lsb_release -is 2020-12-05T19:04:51.256 INFO:teuthology.orchestra.run.smithi185.stdout:Ubuntu 2020-12-05T19:04:51.262 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-12-05T19:04:51.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3b1861acb8137d863184795ccadbd5d0cb026674 2020-12-05T19:04:51.469 INFO:teuthology.orchestra.run.smithi185:> dpkg-query -W -f '${Version}' ceph 2020-12-05T19:04:51.505 INFO:teuthology.orchestra.run.smithi185.stdout:15.2.6-607-g3b1861ac-1bionic 2020-12-05T19:04:51.505 INFO:teuthology.packaging:The installed version of ceph is 15.2.6-607-g3b1861ac-1bionic 2020-12-05T19:04:51.506 INFO:teuthology.task.install:The correct ceph version 15.2.6-607-g3b1861ac-1bionic is installed. 2020-12-05T19:04:51.507 INFO:teuthology.orchestra.run.smithi023:> sudo lsb_release -is 2020-12-05T19:04:51.572 INFO:teuthology.orchestra.run.smithi023.stdout:Ubuntu 2020-12-05T19:04:51.577 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-12-05T19:04:51.578 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3b1861acb8137d863184795ccadbd5d0cb026674 2020-12-05T19:04:51.748 INFO:teuthology.orchestra.run.smithi023:> dpkg-query -W -f '${Version}' ceph 2020-12-05T19:04:51.774 INFO:teuthology.orchestra.run.smithi023.stdout:15.2.6-607-g3b1861ac-1bionic 2020-12-05T19:04:51.775 INFO:teuthology.packaging:The installed version of ceph is 15.2.6-607-g3b1861ac-1bionic 2020-12-05T19:04:51.775 INFO:teuthology.task.install:The correct ceph version 15.2.6-607-g3b1861ac-1bionic is installed. 2020-12-05T19:04:51.776 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-12-05T19:04:51.776 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:04:51.776 INFO:teuthology.orchestra.run.smithi185:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-12-05T19:04:51.787 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-12-05T19:04:51.836 INFO:teuthology.orchestra.run.smithi185.stderr:24+1 records in 2020-12-05T19:04:51.836 INFO:teuthology.orchestra.run.smithi185.stderr:24+1 records out 2020-12-05T19:04:51.837 INFO:teuthology.orchestra.run.smithi185.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.0386144 s, 328 kB/s 2020-12-05T19:04:51.839 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:04:51.839 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-12-05T19:04:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-12-05T19:04:51.857 INFO:teuthology.orchestra.run.smithi023.stderr:24+1 records in 2020-12-05T19:04:51.857 INFO:teuthology.orchestra.run.smithi023.stderr:24+1 records out 2020-12-05T19:04:51.857 INFO:teuthology.orchestra.run.smithi023.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000226231 s, 56.0 MB/s 2020-12-05T19:04:51.858 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-12-05T19:04:51.859 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:04:51.859 INFO:teuthology.orchestra.run.smithi185:> sudo dd of=/usr/bin/daemon-helper 2020-12-05T19:04:51.889 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-12-05T19:04:51.898 INFO:teuthology.orchestra.run.smithi185.stderr:5+1 records in 2020-12-05T19:04:51.898 INFO:teuthology.orchestra.run.smithi185.stderr:5+1 records out 2020-12-05T19:04:51.899 INFO:teuthology.orchestra.run.smithi185.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 0.000101314 s, 30.2 MB/s 2020-12-05T19:04:51.899 INFO:teuthology.orchestra.run.smithi185:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-12-05T19:04:51.959 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:04:51.959 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2020-12-05T19:04:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-12-05T19:04:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:5+1 records in 2020-12-05T19:04:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:5+1 records out 2020-12-05T19:04:51.975 INFO:teuthology.orchestra.run.smithi023.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 0.00010248 s, 29.8 MB/s 2020-12-05T19:04:51.976 INFO:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-12-05T19:04:52.033 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-12-05T19:04:52.033 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:04:52.034 INFO:teuthology.orchestra.run.smithi185:> sudo dd of=/usr/bin/adjust-ulimits 2020-12-05T19:04:52.039 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-12-05T19:04:52.047 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records in 2020-12-05T19:04:52.047 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records out 2020-12-05T19:04:52.047 INFO:teuthology.orchestra.run.smithi185.stderr:299 bytes copied, 3.2118e-05 s, 9.3 MB/s 2020-12-05T19:04:52.048 INFO:teuthology.orchestra.run.smithi185:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-12-05T19:04:52.105 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:04:52.106 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2020-12-05T19:04:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-12-05T19:04:52.120 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-12-05T19:04:52.120 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-12-05T19:04:52.120 INFO:teuthology.orchestra.run.smithi023.stderr:299 bytes copied, 7.3037e-05 s, 4.1 MB/s 2020-12-05T19:04:52.121 INFO:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-12-05T19:04:52.178 INFO:teuthology.run_tasks:Running task ceph... 2020-12-05T19:04:52.228 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-12-05T19:04:52.228 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 777 /var/log/ceph 2020-12-05T19:04:52.231 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 777 /var/log/ceph 2020-12-05T19:04:52.245 INFO:tasks.ceph:Disabling ceph logrotate... 2020-12-05T19:04:52.246 INFO:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/logrotate.d/ceph 2020-12-05T19:04:52.289 INFO:teuthology.orchestra.run.smithi185:> sudo rm -f -- /etc/logrotate.d/ceph 2020-12-05T19:04:52.304 INFO:tasks.ceph:Creating extra log directories... 2020-12-05T19:04:52.304 INFO:teuthology.orchestra.run.smithi023:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-12-05T19:04:52.345 INFO:teuthology.orchestra.run.smithi185:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-12-05T19:04:52.361 INFO:tasks.ceph:Creating ceph cluster c1... 2020-12-05T19:04:52.361 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-12-05T19:04:52.362 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi/5667657', 'branch': 'octopus', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5667657', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'octopus', 'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 1000, 'repo': 'git://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'octopus', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_octopus/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph.git', 'suite_sha1': '3b1861acb8137d863184795ccadbd5d0cb026674', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-30_05:05:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.26405', 'targets': {'smithi185.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjM27pravCWwd4wQHgsJ/bDwVrPOCjgkhWR5PVyCYF56CoY484Bm4Avg063INbNigMP3qdmtLolZjFoUoDcPvjP/qX4K89tPyG41YAU1u4sOKflbBZEhxGh/Rralqy0dEhpZgxDNaBSc+SdLOJOmebRsioCorcAza/21HBrVtNKGdVlbpHT00Kfb7LyKEpAlT4l9/I0zAj+hdW5B55R9tFsXGKThcmFcR2QD4hay4DKwqEvUAuwajvp+MlUd9ftcUxHsLmB+kLbl8xUKZ4l8iaTT9HVO0uBSUXUhAmjBpiQBtmO4clgigeaZNIfNhxILBWg2Ks98OJNi7BKVxHZXzH', 'smithi023.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKicYR1RJs1KLLijJn5XjjMvdd2qAN7whGTrlf9Sts32c4jcSYLivxfaBDl1Z4PYs+admw/twsV80qfYKiRalrC985ss9a8o0NQ5vPQWTaufIwxUIsoUPF8QPg61GrF9pskx3PmsJP0hiN+7ycg290F6UtevZu5XE0vhaRjyshghE/6na6Fc2yGQV6ZGqHkNgLJUlUDqwf/w6JbaR6UoIuQW04UmolcsI6V6YXpri1/e1rd/HWgJ8wtUkRJ78uXHY/hwDz0cWiBtqWAk6wz/u83jFkF2wk7IwxA26ZWBkvoAbcLbhidRkKQUQ8kve6Guk9a19NFBknZ2Cmhv7IigsR'}} 2020-12-05T19:04:52.363 INFO:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-12-05T19:04:52.401 INFO:teuthology.orchestra.run.smithi185:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-12-05T19:04:52.409 INFO:teuthology.orchestra.run.smithi023:> sudo install -d -m0777 -- /var/run/ceph 2020-12-05T19:04:52.449 INFO:teuthology.orchestra.run.smithi185:> sudo install -d -m0777 -- /var/run/ceph 2020-12-05T19:04:52.465 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:04:52.465 INFO:teuthology.orchestra.run.smithi185:> dd if=/scratch_devs of=/dev/stdout 2020-12-05T19:04:52.511 INFO:teuthology.orchestra.run.smithi185.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-12-05T19:04:52.512 INFO:teuthology.orchestra.run.smithi185.stdout:/dev/vg_nvme/lv_1 2020-12-05T19:04:52.512 INFO:teuthology.orchestra.run.smithi185.stdout:/dev/vg_nvme/lv_2 2020-12-05T19:04:52.513 INFO:teuthology.orchestra.run.smithi185.stdout:/dev/vg_nvme/lv_3 2020-12-05T19:04:52.513 INFO:teuthology.orchestra.run.smithi185.stdout:/dev/vg_nvme/lv_4 2020-12-05T19:04:52.513 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records in 2020-12-05T19:04:52.513 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records out 2020-12-05T19:04:52.514 INFO:teuthology.orchestra.run.smithi185.stderr:72 bytes copied, 0.000163077 s, 442 kB/s 2020-12-05T19:04:52.514 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-12-05T19:04:52.514 INFO:teuthology.orchestra.run.smithi185:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-12-05T19:04:52.561 INFO:teuthology.orchestra.run.smithi185.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-12-05T19:04:52.562 INFO:teuthology.orchestra.run.smithi185.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-12-05T19:04:52.562 INFO:teuthology.orchestra.run.smithi185.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-12-05T19:04:52.562 INFO:teuthology.orchestra.run.smithi185.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-12-05T19:04:52.562 INFO:teuthology.orchestra.run.smithi185.stdout:Access: 2020-12-05 19:02:25.850539445 +0000 2020-12-05T19:04:52.562 INFO:teuthology.orchestra.run.smithi185.stdout:Modify: 2020-12-05 19:02:25.846541446 +0000 2020-12-05T19:04:52.563 INFO:teuthology.orchestra.run.smithi185.stdout:Change: 2020-12-05 19:02:25.846541446 +0000 2020-12-05T19:04:52.563 INFO:teuthology.orchestra.run.smithi185.stdout: Birth: - 2020-12-05T19:04:52.571 INFO:teuthology.orchestra.run.smithi185.stderr:1+0 records in 2020-12-05T19:04:52.571 INFO:teuthology.orchestra.run.smithi185.stderr:1+0 records out 2020-12-05T19:04:52.571 INFO:teuthology.orchestra.run.smithi185.stderr:512 bytes copied, 0.000401094 s, 1.3 MB/s 2020-12-05T19:04:52.576 INFO:teuthology.orchestra.run.smithi185:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-12-05T19:04:52.625 INFO:teuthology.orchestra.run.smithi185.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-12-05T19:04:52.625 INFO:teuthology.orchestra.run.smithi185.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-12-05T19:04:52.626 INFO:teuthology.orchestra.run.smithi185.stdout:Device: 6h/6d Inode: 347 Links: 1 2020-12-05T19:04:52.626 INFO:teuthology.orchestra.run.smithi185.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-12-05T19:04:52.626 INFO:teuthology.orchestra.run.smithi185.stdout:Access: 2020-12-05 19:02:26.178375439 +0000 2020-12-05T19:04:52.626 INFO:teuthology.orchestra.run.smithi185.stdout:Modify: 2020-12-05 19:02:26.098415440 +0000 2020-12-05T19:04:52.627 INFO:teuthology.orchestra.run.smithi185.stdout:Change: 2020-12-05 19:02:26.098415440 +0000 2020-12-05T19:04:52.627 INFO:teuthology.orchestra.run.smithi185.stdout: Birth: - 2020-12-05T19:04:52.634 INFO:teuthology.orchestra.run.smithi185.stderr:1+0 records in 2020-12-05T19:04:52.634 INFO:teuthology.orchestra.run.smithi185.stderr:1+0 records out 2020-12-05T19:04:52.634 INFO:teuthology.orchestra.run.smithi185.stderr:512 bytes copied, 0.000345236 s, 1.5 MB/s 2020-12-05T19:04:52.638 INFO:teuthology.orchestra.run.smithi185:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-12-05T19:04:52.685 INFO:teuthology.orchestra.run.smithi185.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-12-05T19:04:52.685 INFO:teuthology.orchestra.run.smithi185.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-12-05T19:04:52.686 INFO:teuthology.orchestra.run.smithi185.stdout:Device: 6h/6d Inode: 353 Links: 1 2020-12-05T19:04:52.686 INFO:teuthology.orchestra.run.smithi185.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-12-05T19:04:52.686 INFO:teuthology.orchestra.run.smithi185.stdout:Access: 2020-12-05 19:02:26.394267433 +0000 2020-12-05T19:04:52.686 INFO:teuthology.orchestra.run.smithi185.stdout:Modify: 2020-12-05 19:02:26.286321436 +0000 2020-12-05T19:04:52.686 INFO:teuthology.orchestra.run.smithi185.stdout:Change: 2020-12-05 19:02:26.286321436 +0000 2020-12-05T19:04:52.687 INFO:teuthology.orchestra.run.smithi185.stdout: Birth: - 2020-12-05T19:04:52.694 INFO:teuthology.orchestra.run.smithi185.stderr:1+0 records in 2020-12-05T19:04:52.694 INFO:teuthology.orchestra.run.smithi185.stderr:1+0 records out 2020-12-05T19:04:52.694 INFO:teuthology.orchestra.run.smithi185.stderr:512 bytes copied, 0.0003806 s, 1.3 MB/s 2020-12-05T19:04:52.697 INFO:teuthology.orchestra.run.smithi185:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-12-05T19:04:52.745 INFO:teuthology.orchestra.run.smithi185.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-12-05T19:04:52.745 INFO:teuthology.orchestra.run.smithi185.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-12-05T19:04:52.746 INFO:teuthology.orchestra.run.smithi185.stdout:Device: 6h/6d Inode: 359 Links: 1 2020-12-05T19:04:52.746 INFO:teuthology.orchestra.run.smithi185.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-12-05T19:04:52.746 INFO:teuthology.orchestra.run.smithi185.stdout:Access: 2020-12-05 19:02:26.610159429 +0000 2020-12-05T19:04:52.746 INFO:teuthology.orchestra.run.smithi185.stdout:Modify: 2020-12-05 19:02:26.514207431 +0000 2020-12-05T19:04:52.747 INFO:teuthology.orchestra.run.smithi185.stdout:Change: 2020-12-05 19:02:26.514207431 +0000 2020-12-05T19:04:52.747 INFO:teuthology.orchestra.run.smithi185.stdout: Birth: - 2020-12-05T19:04:52.755 INFO:teuthology.orchestra.run.smithi185.stderr:1+0 records in 2020-12-05T19:04:52.755 INFO:teuthology.orchestra.run.smithi185.stderr:1+0 records out 2020-12-05T19:04:52.755 INFO:teuthology.orchestra.run.smithi185.stderr:512 bytes copied, 0.00041866 s, 1.2 MB/s 2020-12-05T19:04:52.759 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-12-05T19:04:52.759 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi185.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-12-05T19:04:52.760 INFO:tasks.ceph:Generating config... 2020-12-05T19:04:52.761 INFO:tasks.ceph:[client] debug rgw = 20 2020-12-05T19:04:52.761 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-12-05T19:04:52.762 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-12-05T19:04:52.762 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-12-05T19:04:52.762 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-12-05T19:04:52.763 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-12-05T19:04:52.763 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-12-05T19:04:52.763 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-12-05T19:04:52.763 INFO:tasks.ceph:[client] setgroup = ceph 2020-12-05T19:04:52.764 INFO:tasks.ceph:[client] setuser = ceph 2020-12-05T19:04:52.764 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-12-05T19:04:52.764 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-12-05T19:04:52.765 INFO:tasks.ceph:[mgr] debug ms = 1 2020-12-05T19:04:52.765 INFO:tasks.ceph:[mon] debug mon = 20 2020-12-05T19:04:52.765 INFO:tasks.ceph:[mon] debug ms = 1 2020-12-05T19:04:52.765 INFO:tasks.ceph:[mon] debug paxos = 20 2020-12-05T19:04:52.766 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-12-05T19:04:52.766 INFO:tasks.ceph:[osd] debug filestore = 20 2020-12-05T19:04:52.766 INFO:tasks.ceph:[osd] debug journal = 20 2020-12-05T19:04:52.767 INFO:tasks.ceph:[osd] debug ms = 20 2020-12-05T19:04:52.767 INFO:tasks.ceph:[osd] debug osd = 25 2020-12-05T19:04:52.767 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-12-05T19:04:52.767 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-12-05T19:04:52.768 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-12-05T19:04:52.768 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-12-05T19:04:52.768 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-12-05T19:04:52.769 INFO:tasks.ceph:Setting up c1.mon.a... 2020-12-05T19:04:52.769 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-12-05T19:04:52.834 INFO:teuthology.orchestra.run.smithi185.stdout:creating /etc/ceph/c1.keyring 2020-12-05T19:04:52.836 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-12-05T19:04:52.913 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-12-05T19:04:52.969 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.185')] 2020-12-05T19:04:52.970 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.185 --print /home/ubuntu/cephtest/c1.monmap 2020-12-05T19:04:53.040 INFO:teuthology.orchestra.run.smithi185.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-12-05T19:04:53.040 INFO:teuthology.orchestra.run.smithi185.stdout:monmaptool: generated fsid d376d55a-973d-44aa-82dd-7af675a5d6c5 2020-12-05T19:04:53.040 INFO:teuthology.orchestra.run.smithi185.stdout:epoch 0 2020-12-05T19:04:53.041 INFO:teuthology.orchestra.run.smithi185.stdout:fsid d376d55a-973d-44aa-82dd-7af675a5d6c5 2020-12-05T19:04:53.041 INFO:teuthology.orchestra.run.smithi185.stdout:last_changed 2020-12-05T19:04:53.045061+0000 2020-12-05T19:04:53.041 INFO:teuthology.orchestra.run.smithi185.stdout:created 2020-12-05T19:04:53.045061+0000 2020-12-05T19:04:53.041 INFO:teuthology.orchestra.run.smithi185.stdout:min_mon_release 0 (unknown) 2020-12-05T19:04:53.042 INFO:teuthology.orchestra.run.smithi185.stdout:0: [v2:172.21.15.185:3300/0,v1:172.21.15.185:6789/0] mon.a 2020-12-05T19:04:53.042 INFO:teuthology.orchestra.run.smithi185.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-12-05T19:04:53.043 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID d376d55a-973d-44aa-82dd-7af675a5d6c5... 2020-12-05T19:04:53.044 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-12-05T19:04:53.051 INFO:teuthology.orchestra.run.smithi185:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-12-05T19:04:53.089 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2020-12-05T19:04:53.090 INFO:teuthology.orchestra.run.smithi023.stdout: chdir = "" 2020-12-05T19:04:53.090 INFO:teuthology.orchestra.run.smithi023.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-12-05T19:04:53.090 INFO:teuthology.orchestra.run.smithi023.stdout: auth supported = cephx 2020-12-05T19:04:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: filestore xattr use omap = true 2020-12-05T19:04:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: mon clock drift allowed = 1.000 2020-12-05T19:04:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: osd crush chooseleaf type = 0 2020-12-05T19:04:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: auth debug = true 2020-12-05T19:04:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: ms die on old message = true 2020-12-05T19:04:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: ms die on bug = true 2020-12-05T19:04:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: mon max pg per osd = 10000 # >= luminous 2020-12-05T19:04:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: mon pg warn max object skew = 0 2020-12-05T19:04:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: # disable pg_autoscaler by default for new pools 2020-12-05T19:04:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: osd_pool_default_pg_autoscale_mode = off 2020-12-05T19:04:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: osd pool default size = 2 2020-12-05T19:04:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd allow primary affinity = true 2020-12-05T19:04:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd allow pg remap = true 2020-12-05T19:04:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on legacy crush tunables = false 2020-12-05T19:04:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on crush straw calc version zero = false 2020-12-05T19:04:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on no sortbitwise = false 2020-12-05T19:04:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on osd down out interval zero = false 2020-12-05T19:04:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on too few osds = false 2020-12-05T19:04:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-12-05T19:04:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_pool_no_redundancy = false 2020-12-05T19:04:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.097 INFO:teuthology.orchestra.run.smithi023.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-12-05T19:04:53.097 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.097 INFO:teuthology.orchestra.run.smithi023.stdout: osd default data pool replay window = 5 2020-12-05T19:04:53.097 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.097 INFO:teuthology.orchestra.run.smithi023.stdout: mon allow pool delete = true 2020-12-05T19:04:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: mon cluster log file level = debug 2020-12-05T19:04:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: debug asserts on shutdown = true 2020-12-05T19:04:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: mon host = 172.21.15.185 2020-12-05T19:04:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: osd heartbeat grace = 40 2020-12-05T19:04:53.099 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = d376d55a-973d-44aa-82dd-7af675a5d6c5 2020-12-05T19:04:53.099 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.099 INFO:teuthology.orchestra.run.smithi023.stdout:[osd] 2020-12-05T19:04:53.099 INFO:teuthology.orchestra.run.smithi023.stdout: osd journal size = 100 2020-12-05T19:04:53.099 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: osd scrub load threshold = 5.0 2020-12-05T19:04:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: osd scrub max interval = 600 2020-12-05T19:04:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: osd recover clone overlap = true 2020-12-05T19:04:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: osd recovery max chunk = 1048576 2020-12-05T19:04:53.101 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.101 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug shutdown = true 2020-12-05T19:04:53.101 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug op order = true 2020-12-05T19:04:53.101 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug verify stray on activate = true 2020-12-05T19:04:53.102 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.102 INFO:teuthology.orchestra.run.smithi023.stdout: osd open classes on start = true 2020-12-05T19:04:53.102 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug pg log writeout = true 2020-12-05T19:04:53.102 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.102 INFO:teuthology.orchestra.run.smithi023.stdout: osd deep scrub update digest min age = 30 2020-12-05T19:04:53.103 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.103 INFO:teuthology.orchestra.run.smithi023.stdout: osd map max advance = 10 2020-12-05T19:04:53.103 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.103 INFO:teuthology.orchestra.run.smithi023.stdout: journal zero on create = true 2020-12-05T19:04:53.104 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.104 INFO:teuthology.orchestra.run.smithi023.stdout: filestore ondisk finisher threads = 3 2020-12-05T19:04:53.104 INFO:teuthology.orchestra.run.smithi023.stdout: filestore apply finisher threads = 3 2020-12-05T19:04:53.104 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.104 INFO:teuthology.orchestra.run.smithi023.stdout: bdev debug aio = true 2020-12-05T19:04:53.105 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug misdirected ops = true 2020-12-05T19:04:53.105 INFO:teuthology.orchestra.run.smithi023.stdout: debug filestore = 20 2020-12-05T19:04:53.105 INFO:teuthology.orchestra.run.smithi023.stdout: debug journal = 20 2020-12-05T19:04:53.105 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 20 2020-12-05T19:04:53.105 INFO:teuthology.orchestra.run.smithi023.stdout: debug osd = 25 2020-12-05T19:04:53.106 INFO:teuthology.orchestra.run.smithi023.stdout: osd fast shutdown = False 2020-12-05T19:04:53.106 INFO:teuthology.orchestra.run.smithi023.stdout: rocksdb delete range threshold = 0 2020-12-05T19:04:53.106 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.106 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr] 2020-12-05T19:04:53.106 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2020-12-05T19:04:53.107 INFO:teuthology.orchestra.run.smithi023.stdout: debug mgr = 20 2020-12-05T19:04:53.107 INFO:teuthology.orchestra.run.smithi023.stdout: debug mon = 20 2020-12-05T19:04:53.107 INFO:teuthology.orchestra.run.smithi023.stdout: debug auth = 20 2020-12-05T19:04:53.107 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min pgs per osd = 4 2020-12-05T19:04:53.107 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min bytes per osd = 10 2020-12-05T19:04:53.108 INFO:teuthology.orchestra.run.smithi023.stdout: mgr/telemetry/nag = false 2020-12-05T19:04:53.108 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.108 INFO:teuthology.orchestra.run.smithi023.stdout:[mon] 2020-12-05T19:04:53.108 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2020-12-05T19:04:53.109 INFO:teuthology.orchestra.run.smithi023.stdout: debug mon = 20 2020-12-05T19:04:53.109 INFO:teuthology.orchestra.run.smithi023.stdout: debug paxos = 20 2020-12-05T19:04:53.109 INFO:teuthology.orchestra.run.smithi023.stdout: debug auth = 20 2020-12-05T19:04:53.109 INFO:teuthology.orchestra.run.smithi023.stdout: mon data avail warn = 5 2020-12-05T19:04:53.109 INFO:teuthology.orchestra.run.smithi023.stdout: mon mgr mkfs grace = 240 2020-12-05T19:04:53.110 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min pgs per osd = 4 2020-12-05T19:04:53.110 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd reporter subtree level = osd 2020-12-05T19:04:53.110 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd prime pg temp = true 2020-12-05T19:04:53.110 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min bytes per osd = 10 2020-12-05T19:04:53.110 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd crush smoke test = False 2020-12-05T19:04:53.111 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:04:53.111 INFO:teuthology.orchestra.run.smithi023.stdout:[client] 2020-12-05T19:04:53.111 INFO:teuthology.orchestra.run.smithi023.stdout: rgw cache enabled = true 2020-12-05T19:04:53.111 INFO:teuthology.orchestra.run.smithi023.stdout: rgw enable ops log = true 2020-12-05T19:04:53.111 INFO:teuthology.orchestra.run.smithi023.stdout: rgw enable usage log = true 2020-12-05T19:04:53.112 INFO:teuthology.orchestra.run.smithi023.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-12-05T19:04:53.112 INFO:teuthology.orchestra.run.smithi023.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-12-05T19:04:53.112 INFO:teuthology.orchestra.run.smithi023.stdout: debug rgw = 20 2020-12-05T19:04:53.112 INFO:teuthology.orchestra.run.smithi023.stdout: rgw crypt require ssl = False 2020-12-05T19:04:53.112 INFO:teuthology.orchestra.run.smithi023.stdout: rgw crypt s3 kms backend = testing 2020-12-05T19:04:53.113 INFO:teuthology.orchestra.run.smithi023.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-12-05T19:04:53.113 INFO:teuthology.orchestra.run.smithi023.stdout: rgw curl low speed time = 300 2020-12-05T19:04:53.113 INFO:teuthology.orchestra.run.smithi023.stdout: rgw data log num shards = 4 2020-12-05T19:04:53.113 INFO:teuthology.orchestra.run.smithi023.stdout: rgw md log max shards = 4 2020-12-05T19:04:53.113 INFO:teuthology.orchestra.run.smithi023.stdout: rgw sync log trim interval = 0 2020-12-05T19:04:53.114 INFO:teuthology.orchestra.run.smithi023.stdout: setgroup = ceph 2020-12-05T19:04:53.114 INFO:teuthology.orchestra.run.smithi023.stdout: setuser = ceph 2020-12-05T19:04:53.114 INFO:teuthology.orchestra.run.smithi023.stdout:[mon.a] 2020-12-05T19:04:53.114 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.0] 2020-12-05T19:04:53.114 INFO:teuthology.orchestra.run.smithi023.stdout: osd_max_omap_entries_per_request = 10 2020-12-05T19:04:53.115 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.1] 2020-12-05T19:04:53.115 INFO:teuthology.orchestra.run.smithi023.stdout: osd_max_omap_entries_per_request = 1000 2020-12-05T19:04:53.115 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.2] 2020-12-05T19:04:53.115 INFO:teuthology.orchestra.run.smithi023.stdout: osd_max_omap_entries_per_request = 10000 2020-12-05T19:04:53.117 INFO:teuthology.orchestra.run.smithi185.stdout:[global] 2020-12-05T19:04:53.117 INFO:teuthology.orchestra.run.smithi185.stdout: chdir = "" 2020-12-05T19:04:53.117 INFO:teuthology.orchestra.run.smithi185.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-12-05T19:04:53.117 INFO:teuthology.orchestra.run.smithi185.stdout: auth supported = cephx 2020-12-05T19:04:53.118 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.118 INFO:teuthology.orchestra.run.smithi185.stdout: filestore xattr use omap = true 2020-12-05T19:04:53.118 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.118 INFO:teuthology.orchestra.run.smithi185.stdout: mon clock drift allowed = 1.000 2020-12-05T19:04:53.119 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.119 INFO:teuthology.orchestra.run.smithi185.stdout: osd crush chooseleaf type = 0 2020-12-05T19:04:53.119 INFO:teuthology.orchestra.run.smithi185.stdout: auth debug = true 2020-12-05T19:04:53.119 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.119 INFO:teuthology.orchestra.run.smithi185.stdout: ms die on old message = true 2020-12-05T19:04:53.120 INFO:teuthology.orchestra.run.smithi185.stdout: ms die on bug = true 2020-12-05T19:04:53.120 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.120 INFO:teuthology.orchestra.run.smithi185.stdout: mon max pg per osd = 10000 # >= luminous 2020-12-05T19:04:53.120 INFO:teuthology.orchestra.run.smithi185.stdout: mon pg warn max object skew = 0 2020-12-05T19:04:53.120 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.121 INFO:teuthology.orchestra.run.smithi185.stdout: # disable pg_autoscaler by default for new pools 2020-12-05T19:04:53.121 INFO:teuthology.orchestra.run.smithi185.stdout: osd_pool_default_pg_autoscale_mode = off 2020-12-05T19:04:53.121 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.121 INFO:teuthology.orchestra.run.smithi185.stdout: osd pool default size = 2 2020-12-05T19:04:53.121 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.121 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd allow primary affinity = true 2020-12-05T19:04:53.122 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd allow pg remap = true 2020-12-05T19:04:53.122 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on legacy crush tunables = false 2020-12-05T19:04:53.122 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on crush straw calc version zero = false 2020-12-05T19:04:53.122 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on no sortbitwise = false 2020-12-05T19:04:53.123 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on osd down out interval zero = false 2020-12-05T19:04:53.123 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on too few osds = false 2020-12-05T19:04:53.123 INFO:teuthology.orchestra.run.smithi185.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-12-05T19:04:53.123 INFO:teuthology.orchestra.run.smithi185.stdout: mon_warn_on_pool_no_redundancy = false 2020-12-05T19:04:53.123 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.124 INFO:teuthology.orchestra.run.smithi185.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-12-05T19:04:53.124 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.124 INFO:teuthology.orchestra.run.smithi185.stdout: osd default data pool replay window = 5 2020-12-05T19:04:53.124 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.124 INFO:teuthology.orchestra.run.smithi185.stdout: mon allow pool delete = true 2020-12-05T19:04:53.125 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.125 INFO:teuthology.orchestra.run.smithi185.stdout: mon cluster log file level = debug 2020-12-05T19:04:53.125 INFO:teuthology.orchestra.run.smithi185.stdout: debug asserts on shutdown = true 2020-12-05T19:04:53.125 INFO:teuthology.orchestra.run.smithi185.stdout: mon host = 172.21.15.185 2020-12-05T19:04:53.125 INFO:teuthology.orchestra.run.smithi185.stdout: osd heartbeat grace = 40 2020-12-05T19:04:53.125 INFO:teuthology.orchestra.run.smithi185.stdout: fsid = d376d55a-973d-44aa-82dd-7af675a5d6c5 2020-12-05T19:04:53.126 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.126 INFO:teuthology.orchestra.run.smithi185.stdout:[osd] 2020-12-05T19:04:53.126 INFO:teuthology.orchestra.run.smithi185.stdout: osd journal size = 100 2020-12-05T19:04:53.126 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.126 INFO:teuthology.orchestra.run.smithi185.stdout: osd scrub load threshold = 5.0 2020-12-05T19:04:53.127 INFO:teuthology.orchestra.run.smithi185.stdout: osd scrub max interval = 600 2020-12-05T19:04:53.127 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.127 INFO:teuthology.orchestra.run.smithi185.stdout: osd recover clone overlap = true 2020-12-05T19:04:53.127 INFO:teuthology.orchestra.run.smithi185.stdout: osd recovery max chunk = 1048576 2020-12-05T19:04:53.127 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.128 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug shutdown = true 2020-12-05T19:04:53.128 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug op order = true 2020-12-05T19:04:53.128 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug verify stray on activate = true 2020-12-05T19:04:53.128 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.128 INFO:teuthology.orchestra.run.smithi185.stdout: osd open classes on start = true 2020-12-05T19:04:53.129 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug pg log writeout = true 2020-12-05T19:04:53.129 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.129 INFO:teuthology.orchestra.run.smithi185.stdout: osd deep scrub update digest min age = 30 2020-12-05T19:04:53.129 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.129 INFO:teuthology.orchestra.run.smithi185.stdout: osd map max advance = 10 2020-12-05T19:04:53.129 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.130 INFO:teuthology.orchestra.run.smithi185.stdout: journal zero on create = true 2020-12-05T19:04:53.130 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.130 INFO:teuthology.orchestra.run.smithi185.stdout: filestore ondisk finisher threads = 3 2020-12-05T19:04:53.130 INFO:teuthology.orchestra.run.smithi185.stdout: filestore apply finisher threads = 3 2020-12-05T19:04:53.130 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.131 INFO:teuthology.orchestra.run.smithi185.stdout: bdev debug aio = true 2020-12-05T19:04:53.131 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug misdirected ops = true 2020-12-05T19:04:53.131 INFO:teuthology.orchestra.run.smithi185.stdout: debug filestore = 20 2020-12-05T19:04:53.131 INFO:teuthology.orchestra.run.smithi185.stdout: debug journal = 20 2020-12-05T19:04:53.131 INFO:teuthology.orchestra.run.smithi185.stdout: debug ms = 20 2020-12-05T19:04:53.132 INFO:teuthology.orchestra.run.smithi185.stdout: debug osd = 25 2020-12-05T19:04:53.132 INFO:teuthology.orchestra.run.smithi185.stdout: osd fast shutdown = False 2020-12-05T19:04:53.132 INFO:teuthology.orchestra.run.smithi185.stdout: rocksdb delete range threshold = 0 2020-12-05T19:04:53.132 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.132 INFO:teuthology.orchestra.run.smithi185.stdout:[mgr] 2020-12-05T19:04:53.132 INFO:teuthology.orchestra.run.smithi185.stdout: debug ms = 1 2020-12-05T19:04:53.133 INFO:teuthology.orchestra.run.smithi185.stdout: debug mgr = 20 2020-12-05T19:04:53.133 INFO:teuthology.orchestra.run.smithi185.stdout: debug mon = 20 2020-12-05T19:04:53.133 INFO:teuthology.orchestra.run.smithi185.stdout: debug auth = 20 2020-12-05T19:04:53.133 INFO:teuthology.orchestra.run.smithi185.stdout: mon reweight min pgs per osd = 4 2020-12-05T19:04:53.133 INFO:teuthology.orchestra.run.smithi185.stdout: mon reweight min bytes per osd = 10 2020-12-05T19:04:53.134 INFO:teuthology.orchestra.run.smithi185.stdout: mgr/telemetry/nag = false 2020-12-05T19:04:53.134 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.134 INFO:teuthology.orchestra.run.smithi185.stdout:[mon] 2020-12-05T19:04:53.134 INFO:teuthology.orchestra.run.smithi185.stdout: debug ms = 1 2020-12-05T19:04:53.134 INFO:teuthology.orchestra.run.smithi185.stdout: debug mon = 20 2020-12-05T19:04:53.135 INFO:teuthology.orchestra.run.smithi185.stdout: debug paxos = 20 2020-12-05T19:04:53.135 INFO:teuthology.orchestra.run.smithi185.stdout: debug auth = 20 2020-12-05T19:04:53.135 INFO:teuthology.orchestra.run.smithi185.stdout: mon data avail warn = 5 2020-12-05T19:04:53.135 INFO:teuthology.orchestra.run.smithi185.stdout: mon mgr mkfs grace = 240 2020-12-05T19:04:53.135 INFO:teuthology.orchestra.run.smithi185.stdout: mon reweight min pgs per osd = 4 2020-12-05T19:04:53.136 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd reporter subtree level = osd 2020-12-05T19:04:53.136 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd prime pg temp = true 2020-12-05T19:04:53.136 INFO:teuthology.orchestra.run.smithi185.stdout: mon reweight min bytes per osd = 10 2020-12-05T19:04:53.136 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd crush smoke test = False 2020-12-05T19:04:53.137 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:04:53.137 INFO:teuthology.orchestra.run.smithi185.stdout:[client] 2020-12-05T19:04:53.137 INFO:teuthology.orchestra.run.smithi185.stdout: rgw cache enabled = true 2020-12-05T19:04:53.137 INFO:teuthology.orchestra.run.smithi185.stdout: rgw enable ops log = true 2020-12-05T19:04:53.137 INFO:teuthology.orchestra.run.smithi185.stdout: rgw enable usage log = true 2020-12-05T19:04:53.138 INFO:teuthology.orchestra.run.smithi185.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-12-05T19:04:53.138 INFO:teuthology.orchestra.run.smithi185.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-12-05T19:04:53.138 INFO:teuthology.orchestra.run.smithi185.stdout: debug rgw = 20 2020-12-05T19:04:53.138 INFO:teuthology.orchestra.run.smithi185.stdout: rgw crypt require ssl = False 2020-12-05T19:04:53.138 INFO:teuthology.orchestra.run.smithi185.stdout: rgw crypt s3 kms backend = testing 2020-12-05T19:04:53.139 INFO:teuthology.orchestra.run.smithi185.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-12-05T19:04:53.139 INFO:teuthology.orchestra.run.smithi185.stdout: rgw curl low speed time = 300 2020-12-05T19:04:53.139 INFO:teuthology.orchestra.run.smithi185.stdout: rgw data log num shards = 4 2020-12-05T19:04:53.139 INFO:teuthology.orchestra.run.smithi185.stdout: rgw md log max shards = 4 2020-12-05T19:04:53.139 INFO:teuthology.orchestra.run.smithi185.stdout: rgw sync log trim interval = 0 2020-12-05T19:04:53.140 INFO:teuthology.orchestra.run.smithi185.stdout: setgroup = ceph 2020-12-05T19:04:53.140 INFO:teuthology.orchestra.run.smithi185.stdout: setuser = ceph 2020-12-05T19:04:53.140 INFO:teuthology.orchestra.run.smithi185.stdout:[mon.a] 2020-12-05T19:04:53.140 INFO:teuthology.orchestra.run.smithi185.stdout:[osd.0] 2020-12-05T19:04:53.140 INFO:teuthology.orchestra.run.smithi185.stdout: osd_max_omap_entries_per_request = 10 2020-12-05T19:04:53.140 INFO:teuthology.orchestra.run.smithi185.stdout:[osd.1] 2020-12-05T19:04:53.141 INFO:teuthology.orchestra.run.smithi185.stdout: osd_max_omap_entries_per_request = 1000 2020-12-05T19:04:53.141 INFO:teuthology.orchestra.run.smithi185.stdout:[osd.2] 2020-12-05T19:04:53.141 INFO:teuthology.orchestra.run.smithi185.stdout: osd_max_omap_entries_per_request = 10000 2020-12-05T19:04:53.142 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-12-05T19:04:53.142 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-12-05T19:04:53.227 INFO:tasks.ceph:Copying monmap to all nodes... 2020-12-05T19:04:53.279 DEBUG:teuthology.orchestra.remote:smithi185:/etc/ceph/c1.keyring is 206B 2020-12-05T19:04:53.296 DEBUG:teuthology.orchestra.remote:smithi185:/home/ubuntu/cephtest/c1.monmap is 179B 2020-12-05T19:04:53.307 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi185.front.sepia.ceph.com 2020-12-05T19:04:53.307 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:04:53.308 INFO:teuthology.orchestra.run.smithi185:> sudo dd of=/etc/ceph/c1.keyring 2020-12-05T19:04:53.308 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-12-05T19:04:53.313 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-12-05T19:04:53.322 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records in 2020-12-05T19:04:53.322 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records out 2020-12-05T19:04:53.322 INFO:teuthology.orchestra.run.smithi185.stderr:206 bytes copied, 0.000136251 s, 1.5 MB/s 2020-12-05T19:04:53.323 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-12-05T19:04:53.332 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:04:53.333 INFO:teuthology.orchestra.run.smithi185:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-12-05T19:04:53.379 INFO:teuthology.orchestra.run.smithi185.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-12-05T19:04:53.380 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records in 2020-12-05T19:04:53.381 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records out 2020-12-05T19:04:53.381 INFO:teuthology.orchestra.run.smithi185.stderr:179 bytes copied, 0.000126503 s, 1.4 MB/s 2020-12-05T19:04:53.381 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:04:53.381 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:04:53.382 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/c1.keyring 2020-12-05T19:04:53.382 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-12-05T19:04:53.387 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-12-05T19:04:53.396 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-12-05T19:04:53.397 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-12-05T19:04:53.397 INFO:teuthology.orchestra.run.smithi023.stderr:206 bytes copied, 7.4707e-05 s, 2.8 MB/s 2020-12-05T19:04:53.397 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-12-05T19:04:53.407 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:04:53.408 INFO:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-12-05T19:04:53.456 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-12-05T19:04:53.457 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-12-05T19:04:53.457 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-12-05T19:04:53.458 INFO:teuthology.orchestra.run.smithi023.stderr:179 bytes copied, 7.8394e-05 s, 2.3 MB/s 2020-12-05T19:04:53.458 INFO:tasks.ceph:Setting up mon nodes... 2020-12-05T19:04:53.458 INFO:tasks.ceph:Setting up mgr nodes... 2020-12-05T19:04:53.459 INFO:teuthology.orchestra.run.smithi185:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-12-05T19:04:53.501 INFO:teuthology.orchestra.run.smithi185.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-12-05T19:04:53.503 INFO:tasks.ceph:Setting up mds nodes... 2020-12-05T19:04:53.504 INFO:tasks.ceph_client:Setting up client nodes... 2020-12-05T19:04:53.504 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-12-05T19:04:53.579 INFO:teuthology.orchestra.run.smithi185.stdout:creating /etc/ceph/c1.client.0.keyring 2020-12-05T19:04:53.591 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-12-05T19:04:53.666 INFO:teuthology.orchestra.run.smithi185.stdout:creating /etc/ceph/c1.client.1.keyring 2020-12-05T19:04:53.679 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-12-05T19:04:53.679 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi185.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-12-05T19:04:53.679 INFO:teuthology.orchestra.run.smithi185:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-12-05T19:04:53.737 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-12-05T19:04:53.737 INFO:tasks.ceph:role: c1.osd.0 2020-12-05T19:04:53.738 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi185.front.sepia.ceph.com 2020-12-05T19:04:53.738 INFO:teuthology.orchestra.run.smithi185:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-12-05T19:04:54.262 INFO:teuthology.orchestra.run.smithi185.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-12-05T19:04:54.263 INFO:teuthology.orchestra.run.smithi185.stdout: = sectsz=512 attr=2, projid32bit=1 2020-12-05T19:04:54.263 INFO:teuthology.orchestra.run.smithi185.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-12-05T19:04:54.264 INFO:teuthology.orchestra.run.smithi185.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-12-05T19:04:54.264 INFO:teuthology.orchestra.run.smithi185.stdout: = sunit=0 swidth=0 blks 2020-12-05T19:04:54.264 INFO:teuthology.orchestra.run.smithi185.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-12-05T19:04:54.264 INFO:teuthology.orchestra.run.smithi185.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-12-05T19:04:54.265 INFO:teuthology.orchestra.run.smithi185.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-12-05T19:04:54.265 INFO:teuthology.orchestra.run.smithi185.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-12-05T19:04:54.266 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi185.front.sepia.ceph.com -o noatime 2020-12-05T19:04:54.267 INFO:teuthology.orchestra.run.smithi185:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-12-05T19:04:54.345 INFO:teuthology.orchestra.run.smithi185:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-12-05T19:04:54.357 INFO:teuthology.orchestra.run.smithi185.stderr:sudo: /sbin/restorecon: command not found 2020-12-05T19:04:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-12-05T19:04:54.358 INFO:teuthology.orchestra.run.smithi185:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-12-05T19:04:54.412 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-12-05T19:04:54.412 INFO:tasks.ceph:role: c1.osd.1 2020-12-05T19:04:54.412 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi185.front.sepia.ceph.com 2020-12-05T19:04:54.413 INFO:teuthology.orchestra.run.smithi185:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-12-05T19:04:54.932 INFO:teuthology.orchestra.run.smithi185.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-12-05T19:04:54.933 INFO:teuthology.orchestra.run.smithi185.stdout: = sectsz=512 attr=2, projid32bit=1 2020-12-05T19:04:54.933 INFO:teuthology.orchestra.run.smithi185.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-12-05T19:04:54.933 INFO:teuthology.orchestra.run.smithi185.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-12-05T19:04:54.933 INFO:teuthology.orchestra.run.smithi185.stdout: = sunit=0 swidth=0 blks 2020-12-05T19:04:54.933 INFO:teuthology.orchestra.run.smithi185.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-12-05T19:04:54.934 INFO:teuthology.orchestra.run.smithi185.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-12-05T19:04:54.934 INFO:teuthology.orchestra.run.smithi185.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-12-05T19:04:54.934 INFO:teuthology.orchestra.run.smithi185.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-12-05T19:04:54.935 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi185.front.sepia.ceph.com -o noatime 2020-12-05T19:04:54.935 INFO:teuthology.orchestra.run.smithi185:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-12-05T19:04:54.996 INFO:teuthology.orchestra.run.smithi185:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-12-05T19:04:55.008 INFO:teuthology.orchestra.run.smithi185.stderr:sudo: /sbin/restorecon: command not found 2020-12-05T19:04:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-12-05T19:04:55.009 INFO:teuthology.orchestra.run.smithi185:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-12-05T19:04:55.066 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-12-05T19:04:55.066 INFO:tasks.ceph:role: c1.osd.2 2020-12-05T19:04:55.066 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi185.front.sepia.ceph.com 2020-12-05T19:04:55.067 INFO:teuthology.orchestra.run.smithi185:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-12-05T19:04:55.568 INFO:teuthology.orchestra.run.smithi185.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-12-05T19:04:55.569 INFO:teuthology.orchestra.run.smithi185.stdout: = sectsz=512 attr=2, projid32bit=1 2020-12-05T19:04:55.569 INFO:teuthology.orchestra.run.smithi185.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-12-05T19:04:55.569 INFO:teuthology.orchestra.run.smithi185.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-12-05T19:04:55.569 INFO:teuthology.orchestra.run.smithi185.stdout: = sunit=0 swidth=0 blks 2020-12-05T19:04:55.569 INFO:teuthology.orchestra.run.smithi185.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-12-05T19:04:55.570 INFO:teuthology.orchestra.run.smithi185.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-12-05T19:04:55.570 INFO:teuthology.orchestra.run.smithi185.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-12-05T19:04:55.570 INFO:teuthology.orchestra.run.smithi185.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-12-05T19:04:55.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi185.front.sepia.ceph.com -o noatime 2020-12-05T19:04:55.571 INFO:teuthology.orchestra.run.smithi185:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-12-05T19:04:55.615 INFO:teuthology.orchestra.run.smithi185:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-12-05T19:04:55.669 INFO:teuthology.orchestra.run.smithi185.stderr:sudo: /sbin/restorecon: command not found 2020-12-05T19:04:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-12-05T19:04:55.670 INFO:teuthology.orchestra.run.smithi185:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-12-05T19:04:55.763 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:55.762+0000 7f93ab5f7d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-12-05T19:04:55.765 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:55.766+0000 7f93ab5f7d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-12-05T19:04:55.765 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:55.766+0000 7f93ab5f7d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-12-05T19:04:55.766 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:55.766+0000 7f93ab5f7d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-12-05T19:04:55.766 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:55.766+0000 7f93ab5f7d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-12-05T19:04:55.768 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:55.770+0000 7f93ab5f7d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-12-05T19:04:55.798 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:55.798+0000 7f93ab5f7d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-12-05T19:04:57.694 INFO:teuthology.orchestra.run.smithi185:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-12-05T19:04:57.709 INFO:teuthology.orchestra.run.smithi185:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-12-05T19:04:57.792 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:57.794+0000 7f34feabad80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-12-05T19:04:57.792 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:57.794+0000 7f34feabad80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-12-05T19:04:57.792 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:57.794+0000 7f34feabad80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-12-05T19:04:57.792 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:57.794+0000 7f34feabad80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-12-05T19:04:57.793 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:57.794+0000 7f34feabad80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-12-05T19:04:57.793 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:57.794+0000 7f34feabad80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-12-05T19:04:57.808 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:57.810+0000 7f34feabad80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-12-05T19:04:59.709 INFO:teuthology.orchestra.run.smithi185:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-12-05T19:04:59.766 INFO:teuthology.orchestra.run.smithi185:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-12-05T19:04:59.849 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:59.850+0000 7fbdd2803d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-12-05T19:04:59.849 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:59.850+0000 7fbdd2803d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-12-05T19:04:59.850 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:59.850+0000 7fbdd2803d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-12-05T19:04:59.850 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:59.850+0000 7fbdd2803d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-12-05T19:04:59.850 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:59.850+0000 7fbdd2803d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-12-05T19:04:59.851 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:59.850+0000 7fbdd2803d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-12-05T19:04:59.864 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:04:59.866+0000 7fbdd2803d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-12-05T19:05:01.781 INFO:teuthology.orchestra.run.smithi185:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-12-05T19:05:01.838 INFO:tasks.ceph:Reading keys from all nodes... 2020-12-05T19:05:01.838 INFO:teuthology.orchestra.run.smithi185:> mktemp 2020-12-05T19:05:01.884 INFO:teuthology.orchestra.run.smithi185.stdout:/tmp/tmp.9Q4FmYtJM5 2020-12-05T19:05:01.885 INFO:teuthology.orchestra.run.smithi185:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.9Q4FmYtJM5 2020-12-05T19:05:01.946 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0666 /tmp/tmp.9Q4FmYtJM5 2020-12-05T19:05:02.050 DEBUG:teuthology.orchestra.remote:smithi185:/tmp/tmp.9Q4FmYtJM5 is 56B 2020-12-05T19:05:02.061 INFO:teuthology.orchestra.run.smithi185:> rm -fr /tmp/tmp.9Q4FmYtJM5 2020-12-05T19:05:02.067 INFO:teuthology.orchestra.run.smithi185:> mktemp 2020-12-05T19:05:02.117 INFO:teuthology.orchestra.run.smithi185.stdout:/tmp/tmp.RflCgbDLft 2020-12-05T19:05:02.117 INFO:teuthology.orchestra.run.smithi185:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.RflCgbDLft 2020-12-05T19:05:02.174 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0666 /tmp/tmp.RflCgbDLft 2020-12-05T19:05:02.278 DEBUG:teuthology.orchestra.remote:smithi185:/tmp/tmp.RflCgbDLft is 56B 2020-12-05T19:05:02.289 INFO:teuthology.orchestra.run.smithi185:> rm -fr /tmp/tmp.RflCgbDLft 2020-12-05T19:05:02.295 INFO:teuthology.orchestra.run.smithi185:> mktemp 2020-12-05T19:05:02.344 INFO:teuthology.orchestra.run.smithi185.stdout:/tmp/tmp.ensK51EbOX 2020-12-05T19:05:02.344 INFO:teuthology.orchestra.run.smithi185:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.ensK51EbOX 2020-12-05T19:05:02.402 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0666 /tmp/tmp.ensK51EbOX 2020-12-05T19:05:02.506 DEBUG:teuthology.orchestra.remote:smithi185:/tmp/tmp.ensK51EbOX is 56B 2020-12-05T19:05:02.518 INFO:teuthology.orchestra.run.smithi185:> rm -fr /tmp/tmp.ensK51EbOX 2020-12-05T19:05:02.525 INFO:teuthology.orchestra.run.smithi185:> mktemp 2020-12-05T19:05:02.572 INFO:teuthology.orchestra.run.smithi185.stdout:/tmp/tmp.jF8m1A2pVc 2020-12-05T19:05:02.573 INFO:teuthology.orchestra.run.smithi185:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.jF8m1A2pVc 2020-12-05T19:05:02.629 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0666 /tmp/tmp.jF8m1A2pVc 2020-12-05T19:05:02.734 DEBUG:teuthology.orchestra.remote:smithi185:/tmp/tmp.jF8m1A2pVc is 56B 2020-12-05T19:05:02.744 INFO:teuthology.orchestra.run.smithi185:> rm -fr /tmp/tmp.jF8m1A2pVc 2020-12-05T19:05:02.803 DEBUG:teuthology.orchestra.remote:smithi185:/etc/ceph/c1.client.0.keyring is 59B 2020-12-05T19:05:02.860 DEBUG:teuthology.orchestra.remote:smithi185:/etc/ceph/c1.client.1.keyring is 59B 2020-12-05T19:05:02.871 INFO:tasks.ceph:Adding keys to all mons... 2020-12-05T19:05:02.871 INFO:teuthology.orchestra.run.smithi185:> sudo tee -a /etc/ceph/c1.keyring 2020-12-05T19:05:02.915 INFO:teuthology.orchestra.run.smithi185.stdout:[mgr.x] 2020-12-05T19:05:02.915 INFO:teuthology.orchestra.run.smithi185.stdout: key = AQDV2ctfkcEzHhAAELu4YliiQlkuWtpqxKuQmA== 2020-12-05T19:05:02.916 INFO:teuthology.orchestra.run.smithi185.stdout:[osd.0] 2020-12-05T19:05:02.916 INFO:teuthology.orchestra.run.smithi185.stdout: key = AQDX2ctfPuPJLRAADz4+oGc5XV3+Q598JpNkbA== 2020-12-05T19:05:02.916 INFO:teuthology.orchestra.run.smithi185.stdout:[osd.1] 2020-12-05T19:05:02.916 INFO:teuthology.orchestra.run.smithi185.stdout: key = AQDZ2ctfsBB/LxAAYOPd6Znhc2eHlehMBkPWjQ== 2020-12-05T19:05:02.917 INFO:teuthology.orchestra.run.smithi185.stdout:[osd.2] 2020-12-05T19:05:02.917 INFO:teuthology.orchestra.run.smithi185.stdout: key = AQDb2ctf1rfqMhAA6PDhuyEtnhJxfTyDGxW/Dg== 2020-12-05T19:05:02.917 INFO:teuthology.orchestra.run.smithi185.stdout:[client.0] 2020-12-05T19:05:02.917 INFO:teuthology.orchestra.run.smithi185.stdout: key = AQDV2ctfpqPVIhAAYviFAEeoJqhs+Y6VHGhpPg== 2020-12-05T19:05:02.917 INFO:teuthology.orchestra.run.smithi185.stdout:[client.1] 2020-12-05T19:05:02.918 INFO:teuthology.orchestra.run.smithi185.stdout: key = AQDV2ctf7VoQKBAAosmBnyOx5OCquM4/cUZf8w== 2020-12-05T19:05:02.919 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-12-05T19:05:03.004 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-12-05T19:05:03.087 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-12-05T19:05:03.171 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-12-05T19:05:03.256 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-12-05T19:05:03.343 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-12-05T19:05:03.429 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-12-05T19:05:03.430 INFO:teuthology.orchestra.run.smithi185:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-12-05T19:05:03.485 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-12-05T19:05:03.582 INFO:teuthology.orchestra.run.smithi185:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-12-05T19:05:03.594 INFO:teuthology.orchestra.run.smithi185:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-12-05T19:05:03.644 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-12-05T19:05:03.645 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-12-05T19:05:03.645 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-12-05T19:05:03.688 INFO:tasks.ceph.c1.mon.a:Started 2020-12-05T19:05:03.689 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-12-05T19:05:03.689 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-12-05T19:05:03.689 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-12-05T19:05:03.692 INFO:tasks.ceph.c1.mgr.x:Started 2020-12-05T19:05:03.692 INFO:tasks.ceph:Setting crush tunables to default 2020-12-05T19:05:03.692 INFO:teuthology.orchestra.run.smithi185:> sudo ceph --cluster c1 osd crush tunables default 2020-12-05T19:05:04.093 INFO:teuthology.orchestra.run.smithi185.stderr:adjusted tunables profile to default 2020-12-05T19:05:04.104 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-12-05T19:05:04.104 INFO:teuthology.orchestra.run.smithi185:> mktemp 2020-12-05T19:05:04.109 INFO:teuthology.orchestra.run.smithi185.stdout:/tmp/tmp.7QHgaqFO2b 2020-12-05T19:05:04.109 INFO:teuthology.orchestra.run.smithi185:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.7QHgaqFO2b 2020-12-05T19:05:04.164 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0666 /tmp/tmp.7QHgaqFO2b 2020-12-05T19:05:04.266 DEBUG:teuthology.orchestra.remote:smithi185:/tmp/tmp.7QHgaqFO2b is 37B 2020-12-05T19:05:04.276 INFO:teuthology.orchestra.run.smithi185:> rm -fr /tmp/tmp.7QHgaqFO2b 2020-12-05T19:05:04.282 INFO:teuthology.orchestra.run.smithi185:> mktemp 2020-12-05T19:05:04.332 INFO:teuthology.orchestra.run.smithi185.stdout:/tmp/tmp.fKJvshT3D3 2020-12-05T19:05:04.333 INFO:teuthology.orchestra.run.smithi185:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.fKJvshT3D3 2020-12-05T19:05:04.389 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0666 /tmp/tmp.fKJvshT3D3 2020-12-05T19:05:04.494 DEBUG:teuthology.orchestra.remote:smithi185:/tmp/tmp.fKJvshT3D3 is 37B 2020-12-05T19:05:04.505 INFO:teuthology.orchestra.run.smithi185:> rm -fr /tmp/tmp.fKJvshT3D3 2020-12-05T19:05:04.511 INFO:teuthology.orchestra.run.smithi185:> mktemp 2020-12-05T19:05:04.560 INFO:teuthology.orchestra.run.smithi185.stdout:/tmp/tmp.vEPu9tc3Tn 2020-12-05T19:05:04.560 INFO:teuthology.orchestra.run.smithi185:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.vEPu9tc3Tn 2020-12-05T19:05:04.617 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0666 /tmp/tmp.vEPu9tc3Tn 2020-12-05T19:05:04.722 DEBUG:teuthology.orchestra.remote:smithi185:/tmp/tmp.vEPu9tc3Tn is 37B 2020-12-05T19:05:04.731 INFO:teuthology.orchestra.run.smithi185:> rm -fr /tmp/tmp.vEPu9tc3Tn 2020-12-05T19:05:04.737 INFO:teuthology.orchestra.run.smithi185:> sudo ceph --cluster c1 osd new 2074b5b1-0769-447b-849f-98790a028496 0 2020-12-05T19:05:04.979 INFO:teuthology.orchestra.run.smithi185.stdout:0 2020-12-05T19:05:04.991 INFO:teuthology.orchestra.run.smithi185:> sudo ceph --cluster c1 osd new 721bc876-1de4-4925-9ad8-12157d2e165e 1 2020-12-05T19:05:05.183 INFO:teuthology.orchestra.run.smithi185.stdout:1 2020-12-05T19:05:05.194 INFO:teuthology.orchestra.run.smithi185:> sudo ceph --cluster c1 osd new 1f57e01b-b1ac-4e08-aa51-978c6bb96ecf 2 2020-12-05T19:05:05.394 INFO:teuthology.orchestra.run.smithi185.stdout:2 2020-12-05T19:05:05.411 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-12-05T19:05:05.412 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-12-05T19:05:05.414 INFO:tasks.ceph.c1.osd.0:Started 2020-12-05T19:05:05.414 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-12-05T19:05:05.414 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-12-05T19:05:05.417 INFO:tasks.ceph.c1.osd.1:Started 2020-12-05T19:05:05.417 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-12-05T19:05:05.418 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-12-05T19:05:05.419 INFO:tasks.ceph.c1.osd.2:Started 2020-12-05T19:05:05.420 INFO:tasks.ceph:Waiting for OSDs to come up 2020-12-05T19:05:05.723 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-12-05T19:05:05.896 INFO:teuthology.misc.health.smithi185.stdout: 2020-12-05T19:05:05.897 INFO:teuthology.misc.health.smithi185.stdout:{"epoch":5,"fsid":"d376d55a-973d-44aa-82dd-7af675a5d6c5","created":"2020-12-05T19:05:03.814200+0000","modified":"2020-12-05T19:05:05.395905+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"2074b5b1-0769-447b-849f-98790a028496","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"721bc876-1de4-4925-9ad8-12157d2e165e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1f57e01b-b1ac-4e08-aa51-978c6bb96ecf","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:05.913 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-12-05T19:05:06.016 INFO:tasks.ceph.c1.osd.1.smithi185.stderr:2020-12-05T19:05:06.018+0000 7f1f6a095d80 -1 Falling back to public interface 2020-12-05T19:05:06.025 INFO:tasks.ceph.c1.osd.2.smithi185.stderr:2020-12-05T19:05:06.026+0000 7f3c7c725d80 -1 Falling back to public interface 2020-12-05T19:05:06.035 INFO:tasks.ceph.c1.osd.0.smithi185.stderr:2020-12-05T19:05:06.038+0000 7fa0f3c2dd80 -1 Falling back to public interface 2020-12-05T19:05:06.613 INFO:tasks.ceph.c1.osd.2.smithi185.stderr:2020-12-05T19:05:06.614+0000 7f3c7c725d80 -1 osd.2 0 log_to_monitors {default=true} 2020-12-05T19:05:06.628 INFO:tasks.ceph.c1.osd.1.smithi185.stderr:2020-12-05T19:05:06.630+0000 7f1f6a095d80 -1 osd.1 0 log_to_monitors {default=true} 2020-12-05T19:05:06.635 INFO:tasks.ceph.c1.osd.0.smithi185.stderr:2020-12-05T19:05:06.638+0000 7fa0f3c2dd80 -1 osd.0 0 log_to_monitors {default=true} 2020-12-05T19:05:07.846 INFO:tasks.ceph.c1.osd.0.smithi185.stderr:2020-12-05T19:05:07.846+0000 7fa0dd679700 -1 osd.0 0 waiting for initial osdmap 2020-12-05T19:05:07.846 INFO:tasks.ceph.c1.osd.2.smithi185.stderr:2020-12-05T19:05:07.846+0000 7f3c66171700 -1 osd.2 0 waiting for initial osdmap 2020-12-05T19:05:07.847 INFO:tasks.ceph.c1.osd.1.smithi185.stderr:2020-12-05T19:05:07.846+0000 7f1f53ae1700 -1 osd.1 0 waiting for initial osdmap 2020-12-05T19:05:12.217 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-12-05T19:05:12.479 INFO:teuthology.misc.health.smithi185.stdout: 2020-12-05T19:05:12.480 INFO:teuthology.misc.health.smithi185.stdout:{"epoch":10,"fsid":"d376d55a-973d-44aa-82dd-7af675a5d6c5","created":"2020-12-05T19:05:03.814200+0000","modified":"2020-12-05T19:05:10.868020+0000","last_up_change":"2020-12-05T19:05:08.851331+0000","last_in_change":"2020-12-05T19:05:08.851331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-05T19:05:07.643107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"2074b5b1-0769-447b-849f-98790a028496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6816","nonce":10392},{"type":"v1","addr":"172.21.15.185:6817","nonce":10392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6818","nonce":10392},{"type":"v1","addr":"172.21.15.185:6819","nonce":10392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6822","nonce":10392},{"type":"v1","addr":"172.21.15.185:6823","nonce":10392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6820","nonce":10392},{"type":"v1","addr":"172.21.15.185:6821","nonce":10392}]},"public_addr":"172.21.15.185:6817/10392","cluster_addr":"172.21.15.185:6819/10392","heartbeat_back_addr":"172.21.15.185:6823/10392","heartbeat_front_addr":"172.21.15.185:6821/10392","state":["exists","up"]},{"osd":1,"uuid":"721bc876-1de4-4925-9ad8-12157d2e165e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6800","nonce":10393},{"type":"v1","addr":"172.21.15.185:6801","nonce":10393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6802","nonce":10393},{"type":"v1","addr":"172.21.15.185:6803","nonce":10393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6806","nonce":10393},{"type":"v1","addr":"172.21.15.185:6807","nonce":10393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6804","nonce":10393},{"type":"v1","addr":"172.21.15.185:6805","nonce":10393}]},"public_addr":"172.21.15.185:6801/10393","cluster_addr":"172.21.15.185:6803/10393","heartbeat_back_addr":"172.21.15.185:6807/10393","heartbeat_front_addr":"172.21.15.185:6805/10393","state":["exists","up"]},{"osd":2,"uuid":"1f57e01b-b1ac-4e08-aa51-978c6bb96ecf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6808","nonce":10394},{"type":"v1","addr":"172.21.15.185:6809","nonce":10394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6810","nonce":10394},{"type":"v1","addr":"172.21.15.185:6811","nonce":10394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6814","nonce":10394},{"type":"v1","addr":"172.21.15.185:6815","nonce":10394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6812","nonce":10394},{"type":"v1","addr":"172.21.15.185:6813","nonce":10394}]},"public_addr":"172.21.15.185:6809/10394","cluster_addr":"172.21.15.185:6811/10394","heartbeat_back_addr":"172.21.15.185:6815/10394","heartbeat_front_addr":"172.21.15.185:6813/10394","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:12.498 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-12-05T19:05:12.498 INFO:tasks.ceph:Creating RBD pool 2020-12-05T19:05:12.499 INFO:teuthology.orchestra.run.smithi185:> sudo ceph --cluster c1 osd pool create rbd 8 2020-12-05T19:05:12.891 INFO:teuthology.orchestra.run.smithi185.stderr:pool 'rbd' created 2020-12-05T19:05:12.902 INFO:teuthology.orchestra.run.smithi185:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-12-05T19:05:13.899 INFO:teuthology.orchestra.run.smithi185.stderr:enabled application 'rbd' on pool 'rbd' 2020-12-05T19:05:13.914 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-12-05T19:05:13.915 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-12-05T19:05:13.916 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-12-05T19:05:14.176 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:14.176 INFO:teuthology.orchestra.run.smithi185.stdout:{"epoch":12,"fsid":"d376d55a-973d-44aa-82dd-7af675a5d6c5","created":"2020-12-05T19:05:03.814200+0000","modified":"2020-12-05T19:05:13.885262+0000","last_up_change":"2020-12-05T19:05:08.851331+0000","last_in_change":"2020-12-05T19:05:08.851331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-05T19:05:07.643107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-12-05T19:05:12.773450+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2074b5b1-0769-447b-849f-98790a028496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6816","nonce":10392},{"type":"v1","addr":"172.21.15.185:6817","nonce":10392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6818","nonce":10392},{"type":"v1","addr":"172.21.15.185:6819","nonce":10392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6822","nonce":10392},{"type":"v1","addr":"172.21.15.185:6823","nonce":10392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6820","nonce":10392},{"type":"v1","addr":"172.21.15.185:6821","nonce":10392}]},"public_addr":"172.21.15.185:6817/10392","cluster_addr":"172.21.15.185:6819/10392","heartbeat_back_addr":"172.21.15.185:6823/10392","heartbeat_front_addr":"172.21.15.185:6821/10392","state":["exists","up"]},{"osd":1,"uuid":"721bc876-1de4-4925-9ad8-12157d2e165e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6800","nonce":10393},{"type":"v1","addr":"172.21.15.185:6801","nonce":10393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6802","nonce":10393},{"type":"v1","addr":"172.21.15.185:6803","nonce":10393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6806","nonce":10393},{"type":"v1","addr":"172.21.15.185:6807","nonce":10393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6804","nonce":10393},{"type":"v1","addr":"172.21.15.185:6805","nonce":10393}]},"public_addr":"172.21.15.185:6801/10393","cluster_addr":"172.21.15.185:6803/10393","heartbeat_back_addr":"172.21.15.185:6807/10393","heartbeat_front_addr":"172.21.15.185:6805/10393","state":["exists","up"]},{"osd":2,"uuid":"1f57e01b-b1ac-4e08-aa51-978c6bb96ecf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6808","nonce":10394},{"type":"v1","addr":"172.21.15.185:6809","nonce":10394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6810","nonce":10394},{"type":"v1","addr":"172.21.15.185:6811","nonce":10394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6814","nonce":10394},{"type":"v1","addr":"172.21.15.185:6815","nonce":10394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6812","nonce":10394},{"type":"v1","addr":"172.21.15.185:6813","nonce":10394}]},"public_addr":"172.21.15.185:6809/10394","cluster_addr":"172.21.15.185:6811/10394","heartbeat_back_addr":"172.21.15.185:6815/10394","heartbeat_front_addr":"172.21.15.185:6813/10394","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:14.189 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-05T19:05:07.643107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-05T19:05:12.773450+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-12-05T19:05:14.189 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2020-12-05T19:05:14.470 INFO:teuthology.orchestra.run.smithi185.stdout:pg_num: 1 2020-12-05T19:05:14.483 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2020-12-05T19:05:14.756 INFO:teuthology.orchestra.run.smithi185.stdout:pg_num: 8 2020-12-05T19:05:14.769 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-12-05T19:05:14.769 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-12-05T19:05:14.769 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2020-12-05T19:05:15.075 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:15.088 INFO:teuthology.orchestra.run.smithi185.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6824","nonce":10184},{"type":"v1","addr":"172.21.15.185:6825","nonce":10184}]},"active_addr":"172.21.15.185:6825/10184","active_change":"2020-12-05T19:05:07.610015+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.185:0","nonce":1102917893}]},{"addrvec":[{"type":"v2","addr":"172.21.15.185:0","nonce":4147779715}]},{"addrvec":[{"type":"v2","addr":"172.21.15.185:0","nonce":1734481585}]}]}} 2020-12-05T19:05:15.092 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-12-05T19:05:15.092 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-12-05T19:05:15.093 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-12-05T19:05:15.356 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:15.356 INFO:teuthology.orchestra.run.smithi185.stdout:{"epoch":13,"fsid":"d376d55a-973d-44aa-82dd-7af675a5d6c5","created":"2020-12-05T19:05:03.814200+0000","modified":"2020-12-05T19:05:14.890761+0000","last_up_change":"2020-12-05T19:05:08.851331+0000","last_in_change":"2020-12-05T19:05:08.851331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-05T19:05:07.643107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-12-05T19:05:12.773450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2074b5b1-0769-447b-849f-98790a028496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6816","nonce":10392},{"type":"v1","addr":"172.21.15.185:6817","nonce":10392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6818","nonce":10392},{"type":"v1","addr":"172.21.15.185:6819","nonce":10392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6822","nonce":10392},{"type":"v1","addr":"172.21.15.185:6823","nonce":10392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6820","nonce":10392},{"type":"v1","addr":"172.21.15.185:6821","nonce":10392}]},"public_addr":"172.21.15.185:6817/10392","cluster_addr":"172.21.15.185:6819/10392","heartbeat_back_addr":"172.21.15.185:6823/10392","heartbeat_front_addr":"172.21.15.185:6821/10392","state":["exists","up"]},{"osd":1,"uuid":"721bc876-1de4-4925-9ad8-12157d2e165e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6800","nonce":10393},{"type":"v1","addr":"172.21.15.185:6801","nonce":10393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6802","nonce":10393},{"type":"v1","addr":"172.21.15.185:6803","nonce":10393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6806","nonce":10393},{"type":"v1","addr":"172.21.15.185:6807","nonce":10393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6804","nonce":10393},{"type":"v1","addr":"172.21.15.185:6805","nonce":10393}]},"public_addr":"172.21.15.185:6801/10393","cluster_addr":"172.21.15.185:6803/10393","heartbeat_back_addr":"172.21.15.185:6807/10393","heartbeat_front_addr":"172.21.15.185:6805/10393","state":["exists","up"]},{"osd":2,"uuid":"1f57e01b-b1ac-4e08-aa51-978c6bb96ecf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6808","nonce":10394},{"type":"v1","addr":"172.21.15.185:6809","nonce":10394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6810","nonce":10394},{"type":"v1","addr":"172.21.15.185:6811","nonce":10394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6814","nonce":10394},{"type":"v1","addr":"172.21.15.185:6815","nonce":10394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6812","nonce":10394},{"type":"v1","addr":"172.21.15.185:6813","nonce":10394}]},"public_addr":"172.21.15.185:6809/10394","cluster_addr":"172.21.15.185:6811/10394","heartbeat_back_addr":"172.21.15.185:6815/10394","heartbeat_front_addr":"172.21.15.185:6813/10394","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:15.369 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-12-05T19:05:15.369 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-12-05T19:05:15.640 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:15.640 INFO:teuthology.orchestra.run.smithi185.stdout:{"epoch":13,"fsid":"d376d55a-973d-44aa-82dd-7af675a5d6c5","created":"2020-12-05T19:05:03.814200+0000","modified":"2020-12-05T19:05:14.890761+0000","last_up_change":"2020-12-05T19:05:08.851331+0000","last_in_change":"2020-12-05T19:05:08.851331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-05T19:05:07.643107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-12-05T19:05:12.773450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2074b5b1-0769-447b-849f-98790a028496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6816","nonce":10392},{"type":"v1","addr":"172.21.15.185:6817","nonce":10392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6818","nonce":10392},{"type":"v1","addr":"172.21.15.185:6819","nonce":10392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6822","nonce":10392},{"type":"v1","addr":"172.21.15.185:6823","nonce":10392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6820","nonce":10392},{"type":"v1","addr":"172.21.15.185:6821","nonce":10392}]},"public_addr":"172.21.15.185:6817/10392","cluster_addr":"172.21.15.185:6819/10392","heartbeat_back_addr":"172.21.15.185:6823/10392","heartbeat_front_addr":"172.21.15.185:6821/10392","state":["exists","up"]},{"osd":1,"uuid":"721bc876-1de4-4925-9ad8-12157d2e165e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6800","nonce":10393},{"type":"v1","addr":"172.21.15.185:6801","nonce":10393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6802","nonce":10393},{"type":"v1","addr":"172.21.15.185:6803","nonce":10393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6806","nonce":10393},{"type":"v1","addr":"172.21.15.185:6807","nonce":10393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6804","nonce":10393},{"type":"v1","addr":"172.21.15.185:6805","nonce":10393}]},"public_addr":"172.21.15.185:6801/10393","cluster_addr":"172.21.15.185:6803/10393","heartbeat_back_addr":"172.21.15.185:6807/10393","heartbeat_front_addr":"172.21.15.185:6805/10393","state":["exists","up"]},{"osd":2,"uuid":"1f57e01b-b1ac-4e08-aa51-978c6bb96ecf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6808","nonce":10394},{"type":"v1","addr":"172.21.15.185:6809","nonce":10394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6810","nonce":10394},{"type":"v1","addr":"172.21.15.185:6811","nonce":10394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6814","nonce":10394},{"type":"v1","addr":"172.21.15.185:6815","nonce":10394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.185:6812","nonce":10394},{"type":"v1","addr":"172.21.15.185:6813","nonce":10394}]},"public_addr":"172.21.15.185:6809/10394","cluster_addr":"172.21.15.185:6811/10394","heartbeat_back_addr":"172.21.15.185:6815/10394","heartbeat_front_addr":"172.21.15.185:6813/10394","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:15.653 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2020-12-05T19:05:15.813 INFO:teuthology.orchestra.run.smithi185.stdout:34359738372 2020-12-05T19:05:15.813 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2020-12-05T19:05:15.968 INFO:teuthology.orchestra.run.smithi185.stdout:34359738371 2020-12-05T19:05:15.968 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2020-12-05T19:05:16.126 INFO:teuthology.orchestra.run.smithi185.stdout:34359738371 2020-12-05T19:05:16.127 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-12-05T19:05:16.405 INFO:teuthology.orchestra.run.smithi185.stdout:34359738371 2020-12-05T19:05:16.418 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2020-12-05T19:05:17.419 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-12-05T19:05:17.698 INFO:teuthology.orchestra.run.smithi185.stdout:34359738371 2020-12-05T19:05:17.716 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2020-12-05T19:05:18.717 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-12-05T19:05:19.003 INFO:teuthology.orchestra.run.smithi185.stdout:34359738372 2020-12-05T19:05:19.016 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.0 2020-12-05T19:05:19.017 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2020-12-05T19:05:19.283 INFO:teuthology.orchestra.run.smithi185.stdout:34359738371 2020-12-05T19:05:19.296 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2020-12-05T19:05:19.297 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2020-12-05T19:05:19.563 INFO:teuthology.orchestra.run.smithi185.stdout:34359738371 2020-12-05T19:05:19.576 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.2 2020-12-05T19:05:19.577 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-12-05T19:05:19.577 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-12-05T19:05:19.843 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:19.845 INFO:teuthology.orchestra.run.smithi185.stderr:dumped all 2020-12-05T19:05:19.856 INFO:teuthology.orchestra.run.smithi185.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-12-05T19:05:19.621264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":91932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.725241"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.912300+0000","last_change":"2020-12-05T19:05:13.908994+0000","last_active":"2020-12-05T19:05:13.912300+0000","last_peered":"2020-12-05T19:05:13.912300+0000","last_clean":"2020-12-05T19:05:13.912300+0000","last_became_active":"2020-12-05T19:05:13.908653+0000","last_became_peered":"2020-12-05T19:05:13.908653+0000","last_unstale":"2020-12-05T19:05:13.912300+0000","last_undegraded":"2020-12-05T19:05:13.912300+0000","last_fullsized":"2020-12-05T19:05:13.912300+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.915036+0000","last_change":"2020-12-05T19:05:13.911421+0000","last_active":"2020-12-05T19:05:13.915036+0000","last_peered":"2020-12-05T19:05:13.915036+0000","last_clean":"2020-12-05T19:05:13.915036+0000","last_became_active":"2020-12-05T19:05:13.911099+0000","last_became_peered":"2020-12-05T19:05:13.911099+0000","last_unstale":"2020-12-05T19:05:13.915036+0000","last_undegraded":"2020-12-05T19:05:13.915036+0000","last_fullsized":"2020-12-05T19:05:13.915036+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.912308+0000","last_change":"2020-12-05T19:05:13.908752+0000","last_active":"2020-12-05T19:05:13.912308+0000","last_peered":"2020-12-05T19:05:13.912308+0000","last_clean":"2020-12-05T19:05:13.912308+0000","last_became_active":"2020-12-05T19:05:13.908281+0000","last_became_peered":"2020-12-05T19:05:13.908281+0000","last_unstale":"2020-12-05T19:05:13.912308+0000","last_undegraded":"2020-12-05T19:05:13.912308+0000","last_fullsized":"2020-12-05T19:05:13.912308+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.904867+0000","last_change":"2020-12-05T19:05:13.904867+0000","last_active":"2020-12-05T19:05:13.904867+0000","last_peered":"2020-12-05T19:05:13.904867+0000","last_clean":"2020-12-05T19:05:13.904867+0000","last_became_active":"2020-12-05T19:05:13.904467+0000","last_became_peered":"2020-12-05T19:05:13.904467+0000","last_unstale":"2020-12-05T19:05:13.904867+0000","last_undegraded":"2020-12-05T19:05:13.904867+0000","last_fullsized":"2020-12-05T19:05:13.904867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.901888+0000","last_change":"2020-12-05T19:05:10.523620+0000","last_active":"2020-12-05T19:05:13.901888+0000","last_peered":"2020-12-05T19:05:13.901888+0000","last_clean":"2020-12-05T19:05:13.901888+0000","last_became_active":"2020-12-05T19:05:10.522611+0000","last_became_peered":"2020-12-05T19:05:10.522611+0000","last_unstale":"2020-12-05T19:05:13.901888+0000","last_undegraded":"2020-12-05T19:05:13.901888+0000","last_fullsized":"2020-12-05T19:05:13.901888+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:07.837584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:07.837584+0000","last_clean_scrub_stamp":"2020-12-05T19:05:07.837584+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.912502+0000","last_change":"2020-12-05T19:05:13.906356+0000","last_active":"2020-12-05T19:05:13.912502+0000","last_peered":"2020-12-05T19:05:13.912502+0000","last_clean":"2020-12-05T19:05:13.912502+0000","last_became_active":"2020-12-05T19:05:13.905797+0000","last_became_peered":"2020-12-05T19:05:13.905797+0000","last_unstale":"2020-12-05T19:05:13.912502+0000","last_undegraded":"2020-12-05T19:05:13.912502+0000","last_fullsized":"2020-12-05T19:05:13.912502+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.907848+0000","last_change":"2020-12-05T19:05:13.907848+0000","last_active":"2020-12-05T19:05:13.907848+0000","last_peered":"2020-12-05T19:05:13.907848+0000","last_clean":"2020-12-05T19:05:13.907848+0000","last_became_active":"2020-12-05T19:05:13.907328+0000","last_became_peered":"2020-12-05T19:05:13.907328+0000","last_unstale":"2020-12-05T19:05:13.907848+0000","last_undegraded":"2020-12-05T19:05:13.907848+0000","last_fullsized":"2020-12-05T19:05:13.907848+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.907981+0000","last_change":"2020-12-05T19:05:13.907981+0000","last_active":"2020-12-05T19:05:13.907981+0000","last_peered":"2020-12-05T19:05:13.907981+0000","last_clean":"2020-12-05T19:05:13.907981+0000","last_became_active":"2020-12-05T19:05:13.907556+0000","last_became_peered":"2020-12-05T19:05:13.907556+0000","last_unstale":"2020-12-05T19:05:13.907981+0000","last_undegraded":"2020-12-05T19:05:13.907981+0000","last_fullsized":"2020-12-05T19:05:13.907981+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.915039+0000","last_change":"2020-12-05T19:05:13.911576+0000","last_active":"2020-12-05T19:05:13.915039+0000","last_peered":"2020-12-05T19:05:13.915039+0000","last_clean":"2020-12-05T19:05:13.915039+0000","last_became_active":"2020-12-05T19:05:13.911075+0000","last_became_peered":"2020-12-05T19:05:13.911075+0000","last_unstale":"2020-12-05T19:05:13.915039+0000","last_undegraded":"2020-12-05T19:05:13.915039+0000","last_fullsized":"2020-12-05T19:05:13.915039+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]}]}]}} 2020-12-05T19:05:19.857 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-12-05T19:05:20.122 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:20.124 INFO:teuthology.orchestra.run.smithi185.stderr:dumped all 2020-12-05T19:05:20.135 INFO:teuthology.orchestra.run.smithi185.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-12-05T19:05:19.621264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":91932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.725241"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.912300+0000","last_change":"2020-12-05T19:05:13.908994+0000","last_active":"2020-12-05T19:05:13.912300+0000","last_peered":"2020-12-05T19:05:13.912300+0000","last_clean":"2020-12-05T19:05:13.912300+0000","last_became_active":"2020-12-05T19:05:13.908653+0000","last_became_peered":"2020-12-05T19:05:13.908653+0000","last_unstale":"2020-12-05T19:05:13.912300+0000","last_undegraded":"2020-12-05T19:05:13.912300+0000","last_fullsized":"2020-12-05T19:05:13.912300+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.915036+0000","last_change":"2020-12-05T19:05:13.911421+0000","last_active":"2020-12-05T19:05:13.915036+0000","last_peered":"2020-12-05T19:05:13.915036+0000","last_clean":"2020-12-05T19:05:13.915036+0000","last_became_active":"2020-12-05T19:05:13.911099+0000","last_became_peered":"2020-12-05T19:05:13.911099+0000","last_unstale":"2020-12-05T19:05:13.915036+0000","last_undegraded":"2020-12-05T19:05:13.915036+0000","last_fullsized":"2020-12-05T19:05:13.915036+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.912308+0000","last_change":"2020-12-05T19:05:13.908752+0000","last_active":"2020-12-05T19:05:13.912308+0000","last_peered":"2020-12-05T19:05:13.912308+0000","last_clean":"2020-12-05T19:05:13.912308+0000","last_became_active":"2020-12-05T19:05:13.908281+0000","last_became_peered":"2020-12-05T19:05:13.908281+0000","last_unstale":"2020-12-05T19:05:13.912308+0000","last_undegraded":"2020-12-05T19:05:13.912308+0000","last_fullsized":"2020-12-05T19:05:13.912308+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.904867+0000","last_change":"2020-12-05T19:05:13.904867+0000","last_active":"2020-12-05T19:05:13.904867+0000","last_peered":"2020-12-05T19:05:13.904867+0000","last_clean":"2020-12-05T19:05:13.904867+0000","last_became_active":"2020-12-05T19:05:13.904467+0000","last_became_peered":"2020-12-05T19:05:13.904467+0000","last_unstale":"2020-12-05T19:05:13.904867+0000","last_undegraded":"2020-12-05T19:05:13.904867+0000","last_fullsized":"2020-12-05T19:05:13.904867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.901888+0000","last_change":"2020-12-05T19:05:10.523620+0000","last_active":"2020-12-05T19:05:13.901888+0000","last_peered":"2020-12-05T19:05:13.901888+0000","last_clean":"2020-12-05T19:05:13.901888+0000","last_became_active":"2020-12-05T19:05:10.522611+0000","last_became_peered":"2020-12-05T19:05:10.522611+0000","last_unstale":"2020-12-05T19:05:13.901888+0000","last_undegraded":"2020-12-05T19:05:13.901888+0000","last_fullsized":"2020-12-05T19:05:13.901888+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:07.837584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:07.837584+0000","last_clean_scrub_stamp":"2020-12-05T19:05:07.837584+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.912502+0000","last_change":"2020-12-05T19:05:13.906356+0000","last_active":"2020-12-05T19:05:13.912502+0000","last_peered":"2020-12-05T19:05:13.912502+0000","last_clean":"2020-12-05T19:05:13.912502+0000","last_became_active":"2020-12-05T19:05:13.905797+0000","last_became_peered":"2020-12-05T19:05:13.905797+0000","last_unstale":"2020-12-05T19:05:13.912502+0000","last_undegraded":"2020-12-05T19:05:13.912502+0000","last_fullsized":"2020-12-05T19:05:13.912502+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.907848+0000","last_change":"2020-12-05T19:05:13.907848+0000","last_active":"2020-12-05T19:05:13.907848+0000","last_peered":"2020-12-05T19:05:13.907848+0000","last_clean":"2020-12-05T19:05:13.907848+0000","last_became_active":"2020-12-05T19:05:13.907328+0000","last_became_peered":"2020-12-05T19:05:13.907328+0000","last_unstale":"2020-12-05T19:05:13.907848+0000","last_undegraded":"2020-12-05T19:05:13.907848+0000","last_fullsized":"2020-12-05T19:05:13.907848+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.907981+0000","last_change":"2020-12-05T19:05:13.907981+0000","last_active":"2020-12-05T19:05:13.907981+0000","last_peered":"2020-12-05T19:05:13.907981+0000","last_clean":"2020-12-05T19:05:13.907981+0000","last_became_active":"2020-12-05T19:05:13.907556+0000","last_became_peered":"2020-12-05T19:05:13.907556+0000","last_unstale":"2020-12-05T19:05:13.907981+0000","last_undegraded":"2020-12-05T19:05:13.907981+0000","last_fullsized":"2020-12-05T19:05:13.907981+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:13.915039+0000","last_change":"2020-12-05T19:05:13.911576+0000","last_active":"2020-12-05T19:05:13.915039+0000","last_peered":"2020-12-05T19:05:13.915039+0000","last_clean":"2020-12-05T19:05:13.915039+0000","last_became_active":"2020-12-05T19:05:13.911075+0000","last_became_peered":"2020-12-05T19:05:13.911075+0000","last_unstale":"2020-12-05T19:05:13.915039+0000","last_undegraded":"2020-12-05T19:05:13.915039+0000","last_fullsized":"2020-12-05T19:05:13.915039+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:12.879500+0000","last_clean_scrub_stamp":"2020-12-05T19:05:12.879500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]}]}]}} 2020-12-05T19:05:20.137 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-12-05T19:05:20.137 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-12-05T19:05:20.137 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-12-05T19:05:20.138 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2020-12-05T19:05:20.451 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:20.464 INFO:teuthology.orchestra.run.smithi185.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-12-05T19:05:20.464 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-12-05T19:05:20.465 INFO:teuthology.run_tasks:Running task ceph... 2020-12-05T19:05:20.475 INFO:tasks.ceph:Creating ceph cluster c2... 2020-12-05T19:05:20.476 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-12-05T19:05:20.476 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi/5667657', 'branch': 'octopus', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5667657', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2020-11-30_05:05:02-rgw-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'octopus', 'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 1000, 'repo': 'git://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'octopus', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_octopus/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph.git', 'suite_sha1': '3b1861acb8137d863184795ccadbd5d0cb026674', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3b1861acb8137d863184795ccadbd5d0cb026674', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-30_05:05:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.26405', 'targets': {'smithi185.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjM27pravCWwd4wQHgsJ/bDwVrPOCjgkhWR5PVyCYF56CoY484Bm4Avg063INbNigMP3qdmtLolZjFoUoDcPvjP/qX4K89tPyG41YAU1u4sOKflbBZEhxGh/Rralqy0dEhpZgxDNaBSc+SdLOJOmebRsioCorcAza/21HBrVtNKGdVlbpHT00Kfb7LyKEpAlT4l9/I0zAj+hdW5B55R9tFsXGKThcmFcR2QD4hay4DKwqEvUAuwajvp+MlUd9ftcUxHsLmB+kLbl8xUKZ4l8iaTT9HVO0uBSUXUhAmjBpiQBtmO4clgigeaZNIfNhxILBWg2Ks98OJNi7BKVxHZXzH', 'smithi023.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKicYR1RJs1KLLijJn5XjjMvdd2qAN7whGTrlf9Sts32c4jcSYLivxfaBDl1Z4PYs+admw/twsV80qfYKiRalrC985ss9a8o0NQ5vPQWTaufIwxUIsoUPF8QPg61GrF9pskx3PmsJP0hiN+7ycg290F6UtevZu5XE0vhaRjyshghE/6na6Fc2yGQV6ZGqHkNgLJUlUDqwf/w6JbaR6UoIuQW04UmolcsI6V6YXpri1/e1rd/HWgJ8wtUkRJ78uXHY/hwDz0cWiBtqWAk6wz/u83jFkF2wk7IwxA26ZWBkvoAbcLbhidRkKQUQ8kve6Guk9a19NFBknZ2Cmhv7IigsR'}} 2020-12-05T19:05:20.477 INFO:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-12-05T19:05:20.479 INFO:teuthology.orchestra.run.smithi185:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-12-05T19:05:20.485 INFO:teuthology.orchestra.run.smithi023:> sudo install -d -m0777 -- /var/run/ceph 2020-12-05T19:05:20.529 INFO:teuthology.orchestra.run.smithi185:> sudo install -d -m0777 -- /var/run/ceph 2020-12-05T19:05:20.543 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:05:20.543 INFO:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2020-12-05T19:05:20.587 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-12-05T19:05:20.589 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_1 2020-12-05T19:05:20.589 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_2 2020-12-05T19:05:20.589 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_3 2020-12-05T19:05:20.590 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_4 2020-12-05T19:05:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-12-05T19:05:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-12-05T19:05:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:72 bytes copied, 0.00017675 s, 407 kB/s 2020-12-05T19:05:20.591 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-12-05T19:05:20.591 INFO:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-12-05T19:05:20.638 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-12-05T19:05:20.638 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-12-05T19:05:20.638 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-12-05T19:05:20.639 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-12-05T19:05:20.639 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2020-12-05 19:02:26.302961746 +0000 2020-12-05T19:05:20.639 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2020-12-05 19:02:26.214917744 +0000 2020-12-05T19:05:20.639 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2020-12-05 19:02:26.214917744 +0000 2020-12-05T19:05:20.639 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2020-12-05T19:05:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2020-12-05T19:05:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2020-12-05T19:05:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000607901 s, 842 kB/s 2020-12-05T19:05:20.652 INFO:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-12-05T19:05:20.702 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-12-05T19:05:20.702 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-12-05T19:05:20.703 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 347 Links: 1 2020-12-05T19:05:20.703 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-12-05T19:05:20.703 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2020-12-05 19:02:26.547083752 +0000 2020-12-05T19:05:20.703 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2020-12-05 19:02:26.447033749 +0000 2020-12-05T19:05:20.703 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2020-12-05 19:02:26.447033749 +0000 2020-12-05T19:05:20.704 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2020-12-05T19:05:20.711 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2020-12-05T19:05:20.712 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2020-12-05T19:05:20.712 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000509039 s, 1.0 MB/s 2020-12-05T19:05:20.716 INFO:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-12-05T19:05:20.766 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-12-05T19:05:20.766 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-12-05T19:05:20.766 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 353 Links: 1 2020-12-05T19:05:20.767 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-12-05T19:05:20.767 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2020-12-05 19:02:26.787203757 +0000 2020-12-05T19:05:20.767 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2020-12-05 19:02:26.695157755 +0000 2020-12-05T19:05:20.767 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2020-12-05 19:02:26.695157755 +0000 2020-12-05T19:05:20.767 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2020-12-05T19:05:20.776 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2020-12-05T19:05:20.776 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2020-12-05T19:05:20.776 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000493001 s, 1.0 MB/s 2020-12-05T19:05:20.780 INFO:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-12-05T19:05:20.830 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-12-05T19:05:20.830 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-12-05T19:05:20.830 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 359 Links: 1 2020-12-05T19:05:20.830 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-12-05T19:05:20.831 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2020-12-05 19:02:26.907263759 +0000 2020-12-05T19:05:20.831 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2020-12-05 19:02:26.903261760 +0000 2020-12-05T19:05:20.831 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2020-12-05 19:02:26.903261760 +0000 2020-12-05T19:05:20.831 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2020-12-05T19:05:20.840 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2020-12-05T19:05:20.840 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2020-12-05T19:05:20.840 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000503748 s, 1.0 MB/s 2020-12-05T19:05:20.844 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-12-05T19:05:20.844 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi023.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-12-05T19:05:20.845 INFO:tasks.ceph:Generating config... 2020-12-05T19:05:20.846 INFO:tasks.ceph:[client] debug rgw = 20 2020-12-05T19:05:20.846 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-12-05T19:05:20.846 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-12-05T19:05:20.847 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-12-05T19:05:20.847 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-12-05T19:05:20.847 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-12-05T19:05:20.848 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-12-05T19:05:20.848 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-12-05T19:05:20.848 INFO:tasks.ceph:[client] setgroup = ceph 2020-12-05T19:05:20.848 INFO:tasks.ceph:[client] setuser = ceph 2020-12-05T19:05:20.849 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-12-05T19:05:20.849 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-12-05T19:05:20.849 INFO:tasks.ceph:[mgr] debug ms = 1 2020-12-05T19:05:20.850 INFO:tasks.ceph:[mon] debug mon = 20 2020-12-05T19:05:20.850 INFO:tasks.ceph:[mon] debug ms = 1 2020-12-05T19:05:20.850 INFO:tasks.ceph:[mon] debug paxos = 20 2020-12-05T19:05:20.851 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-12-05T19:05:20.851 INFO:tasks.ceph:[osd] debug filestore = 20 2020-12-05T19:05:20.851 INFO:tasks.ceph:[osd] debug journal = 20 2020-12-05T19:05:20.851 INFO:tasks.ceph:[osd] debug ms = 20 2020-12-05T19:05:20.852 INFO:tasks.ceph:[osd] debug osd = 25 2020-12-05T19:05:20.852 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-12-05T19:05:20.852 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-12-05T19:05:20.853 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-12-05T19:05:20.853 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-12-05T19:05:20.853 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-12-05T19:05:20.854 INFO:tasks.ceph:Setting up c2.mon.a... 2020-12-05T19:05:20.854 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-12-05T19:05:20.919 INFO:teuthology.orchestra.run.smithi023.stdout:creating /etc/ceph/c2.keyring 2020-12-05T19:05:20.921 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-12-05T19:05:20.998 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-12-05T19:05:21.054 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.23')] 2020-12-05T19:05:21.054 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.23 --print /home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool: generated fsid 5bc95e66-3ac9-4ec0-832b-f0600f6214f8 2020-12-05T19:05:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 0 2020-12-05T19:05:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:fsid 5bc95e66-3ac9-4ec0-832b-f0600f6214f8 2020-12-05T19:05:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:last_changed 2020-12-05T19:05:21.127713+0000 2020-12-05T19:05:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:created 2020-12-05T19:05:21.127713+0000 2020-12-05T19:05:21.125 INFO:teuthology.orchestra.run.smithi023.stdout:min_mon_release 0 (unknown) 2020-12-05T19:05:21.125 INFO:teuthology.orchestra.run.smithi023.stdout:0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2020-12-05T19:05:21.125 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-12-05T19:05:21.126 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 5bc95e66-3ac9-4ec0-832b-f0600f6214f8... 2020-12-05T19:05:21.127 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-12-05T19:05:21.169 INFO:teuthology.orchestra.run.smithi185:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-12-05T19:05:21.200 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2020-12-05T19:05:21.200 INFO:teuthology.orchestra.run.smithi023.stdout: chdir = "" 2020-12-05T19:05:21.200 INFO:teuthology.orchestra.run.smithi023.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-12-05T19:05:21.201 INFO:teuthology.orchestra.run.smithi023.stdout: auth supported = cephx 2020-12-05T19:05:21.201 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.201 INFO:teuthology.orchestra.run.smithi023.stdout: filestore xattr use omap = true 2020-12-05T19:05:21.201 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.201 INFO:teuthology.orchestra.run.smithi023.stdout: mon clock drift allowed = 1.000 2020-12-05T19:05:21.202 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.202 INFO:teuthology.orchestra.run.smithi023.stdout: osd crush chooseleaf type = 0 2020-12-05T19:05:21.202 INFO:teuthology.orchestra.run.smithi023.stdout: auth debug = true 2020-12-05T19:05:21.202 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.203 INFO:teuthology.orchestra.run.smithi023.stdout: ms die on old message = true 2020-12-05T19:05:21.203 INFO:teuthology.orchestra.run.smithi023.stdout: ms die on bug = true 2020-12-05T19:05:21.203 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.203 INFO:teuthology.orchestra.run.smithi023.stdout: mon max pg per osd = 10000 # >= luminous 2020-12-05T19:05:21.203 INFO:teuthology.orchestra.run.smithi023.stdout: mon pg warn max object skew = 0 2020-12-05T19:05:21.204 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.204 INFO:teuthology.orchestra.run.smithi023.stdout: # disable pg_autoscaler by default for new pools 2020-12-05T19:05:21.204 INFO:teuthology.orchestra.run.smithi023.stdout: osd_pool_default_pg_autoscale_mode = off 2020-12-05T19:05:21.204 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.204 INFO:teuthology.orchestra.run.smithi023.stdout: osd pool default size = 2 2020-12-05T19:05:21.205 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.205 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd allow primary affinity = true 2020-12-05T19:05:21.205 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd allow pg remap = true 2020-12-05T19:05:21.205 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on legacy crush tunables = false 2020-12-05T19:05:21.205 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on crush straw calc version zero = false 2020-12-05T19:05:21.206 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on no sortbitwise = false 2020-12-05T19:05:21.206 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on osd down out interval zero = false 2020-12-05T19:05:21.206 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on too few osds = false 2020-12-05T19:05:21.206 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-12-05T19:05:21.207 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_pool_no_redundancy = false 2020-12-05T19:05:21.207 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.207 INFO:teuthology.orchestra.run.smithi023.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-12-05T19:05:21.207 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.207 INFO:teuthology.orchestra.run.smithi023.stdout: osd default data pool replay window = 5 2020-12-05T19:05:21.208 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.208 INFO:teuthology.orchestra.run.smithi023.stdout: mon allow pool delete = true 2020-12-05T19:05:21.208 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.208 INFO:teuthology.orchestra.run.smithi023.stdout: mon cluster log file level = debug 2020-12-05T19:05:21.208 INFO:teuthology.orchestra.run.smithi023.stdout: debug asserts on shutdown = true 2020-12-05T19:05:21.209 INFO:teuthology.orchestra.run.smithi023.stdout: mon host = 172.21.15.23 2020-12-05T19:05:21.209 INFO:teuthology.orchestra.run.smithi023.stdout: osd heartbeat grace = 40 2020-12-05T19:05:21.209 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = 5bc95e66-3ac9-4ec0-832b-f0600f6214f8 2020-12-05T19:05:21.209 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.209 INFO:teuthology.orchestra.run.smithi023.stdout:[osd] 2020-12-05T19:05:21.210 INFO:teuthology.orchestra.run.smithi023.stdout: osd journal size = 100 2020-12-05T19:05:21.210 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.210 INFO:teuthology.orchestra.run.smithi023.stdout: osd scrub load threshold = 5.0 2020-12-05T19:05:21.210 INFO:teuthology.orchestra.run.smithi023.stdout: osd scrub max interval = 600 2020-12-05T19:05:21.211 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.211 INFO:teuthology.orchestra.run.smithi023.stdout: osd recover clone overlap = true 2020-12-05T19:05:21.211 INFO:teuthology.orchestra.run.smithi023.stdout: osd recovery max chunk = 1048576 2020-12-05T19:05:21.211 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.212 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug shutdown = true 2020-12-05T19:05:21.212 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug op order = true 2020-12-05T19:05:21.212 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug verify stray on activate = true 2020-12-05T19:05:21.212 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.212 INFO:teuthology.orchestra.run.smithi023.stdout: osd open classes on start = true 2020-12-05T19:05:21.213 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug pg log writeout = true 2020-12-05T19:05:21.213 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.213 INFO:teuthology.orchestra.run.smithi023.stdout: osd deep scrub update digest min age = 30 2020-12-05T19:05:21.213 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.213 INFO:teuthology.orchestra.run.smithi023.stdout: osd map max advance = 10 2020-12-05T19:05:21.214 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.214 INFO:teuthology.orchestra.run.smithi023.stdout: journal zero on create = true 2020-12-05T19:05:21.214 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.214 INFO:teuthology.orchestra.run.smithi023.stdout: filestore ondisk finisher threads = 3 2020-12-05T19:05:21.215 INFO:teuthology.orchestra.run.smithi023.stdout: filestore apply finisher threads = 3 2020-12-05T19:05:21.215 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.215 INFO:teuthology.orchestra.run.smithi023.stdout: bdev debug aio = true 2020-12-05T19:05:21.215 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug misdirected ops = true 2020-12-05T19:05:21.215 INFO:teuthology.orchestra.run.smithi023.stdout: debug filestore = 20 2020-12-05T19:05:21.216 INFO:teuthology.orchestra.run.smithi023.stdout: debug journal = 20 2020-12-05T19:05:21.216 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 20 2020-12-05T19:05:21.216 INFO:teuthology.orchestra.run.smithi023.stdout: debug osd = 25 2020-12-05T19:05:21.216 INFO:teuthology.orchestra.run.smithi023.stdout: osd fast shutdown = False 2020-12-05T19:05:21.216 INFO:teuthology.orchestra.run.smithi023.stdout: rocksdb delete range threshold = 0 2020-12-05T19:05:21.217 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.217 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr] 2020-12-05T19:05:21.217 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2020-12-05T19:05:21.217 INFO:teuthology.orchestra.run.smithi023.stdout: debug mgr = 20 2020-12-05T19:05:21.217 INFO:teuthology.orchestra.run.smithi023.stdout: debug mon = 20 2020-12-05T19:05:21.218 INFO:teuthology.orchestra.run.smithi023.stdout: debug auth = 20 2020-12-05T19:05:21.218 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min pgs per osd = 4 2020-12-05T19:05:21.218 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min bytes per osd = 10 2020-12-05T19:05:21.218 INFO:teuthology.orchestra.run.smithi023.stdout: mgr/telemetry/nag = false 2020-12-05T19:05:21.219 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.219 INFO:teuthology.orchestra.run.smithi023.stdout:[mon] 2020-12-05T19:05:21.219 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2020-12-05T19:05:21.219 INFO:teuthology.orchestra.run.smithi023.stdout: debug mon = 20 2020-12-05T19:05:21.219 INFO:teuthology.orchestra.run.smithi023.stdout: debug paxos = 20 2020-12-05T19:05:21.220 INFO:teuthology.orchestra.run.smithi023.stdout: debug auth = 20 2020-12-05T19:05:21.220 INFO:teuthology.orchestra.run.smithi023.stdout: mon data avail warn = 5 2020-12-05T19:05:21.220 INFO:teuthology.orchestra.run.smithi023.stdout: mon mgr mkfs grace = 240 2020-12-05T19:05:21.220 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min pgs per osd = 4 2020-12-05T19:05:21.220 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd reporter subtree level = osd 2020-12-05T19:05:21.221 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd prime pg temp = true 2020-12-05T19:05:21.221 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min bytes per osd = 10 2020-12-05T19:05:21.221 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd crush smoke test = False 2020-12-05T19:05:21.221 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:21.221 INFO:teuthology.orchestra.run.smithi023.stdout:[client] 2020-12-05T19:05:21.222 INFO:teuthology.orchestra.run.smithi023.stdout: rgw cache enabled = true 2020-12-05T19:05:21.222 INFO:teuthology.orchestra.run.smithi023.stdout: rgw enable ops log = true 2020-12-05T19:05:21.222 INFO:teuthology.orchestra.run.smithi023.stdout: rgw enable usage log = true 2020-12-05T19:05:21.222 INFO:teuthology.orchestra.run.smithi023.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-12-05T19:05:21.223 INFO:teuthology.orchestra.run.smithi023.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-12-05T19:05:21.223 INFO:teuthology.orchestra.run.smithi023.stdout: debug rgw = 20 2020-12-05T19:05:21.223 INFO:teuthology.orchestra.run.smithi023.stdout: rgw crypt require ssl = False 2020-12-05T19:05:21.223 INFO:teuthology.orchestra.run.smithi023.stdout: rgw crypt s3 kms backend = testing 2020-12-05T19:05:21.223 INFO:teuthology.orchestra.run.smithi023.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-12-05T19:05:21.224 INFO:teuthology.orchestra.run.smithi023.stdout: rgw curl low speed time = 300 2020-12-05T19:05:21.224 INFO:teuthology.orchestra.run.smithi023.stdout: rgw data log num shards = 4 2020-12-05T19:05:21.224 INFO:teuthology.orchestra.run.smithi023.stdout: rgw md log max shards = 4 2020-12-05T19:05:21.224 INFO:teuthology.orchestra.run.smithi023.stdout: rgw sync log trim interval = 0 2020-12-05T19:05:21.224 INFO:teuthology.orchestra.run.smithi023.stdout: setgroup = ceph 2020-12-05T19:05:21.225 INFO:teuthology.orchestra.run.smithi023.stdout: setuser = ceph 2020-12-05T19:05:21.225 INFO:teuthology.orchestra.run.smithi023.stdout:[mon.a] 2020-12-05T19:05:21.225 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.0] 2020-12-05T19:05:21.225 INFO:teuthology.orchestra.run.smithi023.stdout: osd_max_omap_entries_per_request = 10 2020-12-05T19:05:21.225 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.1] 2020-12-05T19:05:21.226 INFO:teuthology.orchestra.run.smithi023.stdout: osd_max_omap_entries_per_request = 1000 2020-12-05T19:05:21.226 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.2] 2020-12-05T19:05:21.226 INFO:teuthology.orchestra.run.smithi023.stdout: osd_max_omap_entries_per_request = 10000 2020-12-05T19:05:21.227 INFO:teuthology.orchestra.run.smithi185.stdout:[global] 2020-12-05T19:05:21.228 INFO:teuthology.orchestra.run.smithi185.stdout: chdir = "" 2020-12-05T19:05:21.228 INFO:teuthology.orchestra.run.smithi185.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-12-05T19:05:21.228 INFO:teuthology.orchestra.run.smithi185.stdout: auth supported = cephx 2020-12-05T19:05:21.228 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.229 INFO:teuthology.orchestra.run.smithi185.stdout: filestore xattr use omap = true 2020-12-05T19:05:21.229 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.229 INFO:teuthology.orchestra.run.smithi185.stdout: mon clock drift allowed = 1.000 2020-12-05T19:05:21.229 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.229 INFO:teuthology.orchestra.run.smithi185.stdout: osd crush chooseleaf type = 0 2020-12-05T19:05:21.230 INFO:teuthology.orchestra.run.smithi185.stdout: auth debug = true 2020-12-05T19:05:21.230 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.230 INFO:teuthology.orchestra.run.smithi185.stdout: ms die on old message = true 2020-12-05T19:05:21.230 INFO:teuthology.orchestra.run.smithi185.stdout: ms die on bug = true 2020-12-05T19:05:21.230 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.231 INFO:teuthology.orchestra.run.smithi185.stdout: mon max pg per osd = 10000 # >= luminous 2020-12-05T19:05:21.231 INFO:teuthology.orchestra.run.smithi185.stdout: mon pg warn max object skew = 0 2020-12-05T19:05:21.231 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.231 INFO:teuthology.orchestra.run.smithi185.stdout: # disable pg_autoscaler by default for new pools 2020-12-05T19:05:21.231 INFO:teuthology.orchestra.run.smithi185.stdout: osd_pool_default_pg_autoscale_mode = off 2020-12-05T19:05:21.232 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.232 INFO:teuthology.orchestra.run.smithi185.stdout: osd pool default size = 2 2020-12-05T19:05:21.232 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.232 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd allow primary affinity = true 2020-12-05T19:05:21.233 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd allow pg remap = true 2020-12-05T19:05:21.233 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on legacy crush tunables = false 2020-12-05T19:05:21.233 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on crush straw calc version zero = false 2020-12-05T19:05:21.233 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on no sortbitwise = false 2020-12-05T19:05:21.233 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on osd down out interval zero = false 2020-12-05T19:05:21.234 INFO:teuthology.orchestra.run.smithi185.stdout: mon warn on too few osds = false 2020-12-05T19:05:21.234 INFO:teuthology.orchestra.run.smithi185.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-12-05T19:05:21.234 INFO:teuthology.orchestra.run.smithi185.stdout: mon_warn_on_pool_no_redundancy = false 2020-12-05T19:05:21.234 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.234 INFO:teuthology.orchestra.run.smithi185.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-12-05T19:05:21.235 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.235 INFO:teuthology.orchestra.run.smithi185.stdout: osd default data pool replay window = 5 2020-12-05T19:05:21.235 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.235 INFO:teuthology.orchestra.run.smithi185.stdout: mon allow pool delete = true 2020-12-05T19:05:21.235 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.236 INFO:teuthology.orchestra.run.smithi185.stdout: mon cluster log file level = debug 2020-12-05T19:05:21.236 INFO:teuthology.orchestra.run.smithi185.stdout: debug asserts on shutdown = true 2020-12-05T19:05:21.236 INFO:teuthology.orchestra.run.smithi185.stdout: mon host = 172.21.15.23 2020-12-05T19:05:21.236 INFO:teuthology.orchestra.run.smithi185.stdout: osd heartbeat grace = 40 2020-12-05T19:05:21.237 INFO:teuthology.orchestra.run.smithi185.stdout: fsid = 5bc95e66-3ac9-4ec0-832b-f0600f6214f8 2020-12-05T19:05:21.237 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.237 INFO:teuthology.orchestra.run.smithi185.stdout:[osd] 2020-12-05T19:05:21.237 INFO:teuthology.orchestra.run.smithi185.stdout: osd journal size = 100 2020-12-05T19:05:21.237 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.238 INFO:teuthology.orchestra.run.smithi185.stdout: osd scrub load threshold = 5.0 2020-12-05T19:05:21.238 INFO:teuthology.orchestra.run.smithi185.stdout: osd scrub max interval = 600 2020-12-05T19:05:21.238 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.238 INFO:teuthology.orchestra.run.smithi185.stdout: osd recover clone overlap = true 2020-12-05T19:05:21.238 INFO:teuthology.orchestra.run.smithi185.stdout: osd recovery max chunk = 1048576 2020-12-05T19:05:21.239 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.239 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug shutdown = true 2020-12-05T19:05:21.239 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug op order = true 2020-12-05T19:05:21.239 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug verify stray on activate = true 2020-12-05T19:05:21.240 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.240 INFO:teuthology.orchestra.run.smithi185.stdout: osd open classes on start = true 2020-12-05T19:05:21.240 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug pg log writeout = true 2020-12-05T19:05:21.240 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.240 INFO:teuthology.orchestra.run.smithi185.stdout: osd deep scrub update digest min age = 30 2020-12-05T19:05:21.241 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.241 INFO:teuthology.orchestra.run.smithi185.stdout: osd map max advance = 10 2020-12-05T19:05:21.241 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.241 INFO:teuthology.orchestra.run.smithi185.stdout: journal zero on create = true 2020-12-05T19:05:21.241 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.242 INFO:teuthology.orchestra.run.smithi185.stdout: filestore ondisk finisher threads = 3 2020-12-05T19:05:21.242 INFO:teuthology.orchestra.run.smithi185.stdout: filestore apply finisher threads = 3 2020-12-05T19:05:21.242 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.242 INFO:teuthology.orchestra.run.smithi185.stdout: bdev debug aio = true 2020-12-05T19:05:21.242 INFO:teuthology.orchestra.run.smithi185.stdout: osd debug misdirected ops = true 2020-12-05T19:05:21.243 INFO:teuthology.orchestra.run.smithi185.stdout: debug filestore = 20 2020-12-05T19:05:21.243 INFO:teuthology.orchestra.run.smithi185.stdout: debug journal = 20 2020-12-05T19:05:21.243 INFO:teuthology.orchestra.run.smithi185.stdout: debug ms = 20 2020-12-05T19:05:21.243 INFO:teuthology.orchestra.run.smithi185.stdout: debug osd = 25 2020-12-05T19:05:21.244 INFO:teuthology.orchestra.run.smithi185.stdout: osd fast shutdown = False 2020-12-05T19:05:21.244 INFO:teuthology.orchestra.run.smithi185.stdout: rocksdb delete range threshold = 0 2020-12-05T19:05:21.244 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.244 INFO:teuthology.orchestra.run.smithi185.stdout:[mgr] 2020-12-05T19:05:21.244 INFO:teuthology.orchestra.run.smithi185.stdout: debug ms = 1 2020-12-05T19:05:21.245 INFO:teuthology.orchestra.run.smithi185.stdout: debug mgr = 20 2020-12-05T19:05:21.245 INFO:teuthology.orchestra.run.smithi185.stdout: debug mon = 20 2020-12-05T19:05:21.245 INFO:teuthology.orchestra.run.smithi185.stdout: debug auth = 20 2020-12-05T19:05:21.245 INFO:teuthology.orchestra.run.smithi185.stdout: mon reweight min pgs per osd = 4 2020-12-05T19:05:21.245 INFO:teuthology.orchestra.run.smithi185.stdout: mon reweight min bytes per osd = 10 2020-12-05T19:05:21.246 INFO:teuthology.orchestra.run.smithi185.stdout: mgr/telemetry/nag = false 2020-12-05T19:05:21.246 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.246 INFO:teuthology.orchestra.run.smithi185.stdout:[mon] 2020-12-05T19:05:21.246 INFO:teuthology.orchestra.run.smithi185.stdout: debug ms = 1 2020-12-05T19:05:21.247 INFO:teuthology.orchestra.run.smithi185.stdout: debug mon = 20 2020-12-05T19:05:21.247 INFO:teuthology.orchestra.run.smithi185.stdout: debug paxos = 20 2020-12-05T19:05:21.247 INFO:teuthology.orchestra.run.smithi185.stdout: debug auth = 20 2020-12-05T19:05:21.247 INFO:teuthology.orchestra.run.smithi185.stdout: mon data avail warn = 5 2020-12-05T19:05:21.247 INFO:teuthology.orchestra.run.smithi185.stdout: mon mgr mkfs grace = 240 2020-12-05T19:05:21.248 INFO:teuthology.orchestra.run.smithi185.stdout: mon reweight min pgs per osd = 4 2020-12-05T19:05:21.248 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd reporter subtree level = osd 2020-12-05T19:05:21.248 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd prime pg temp = true 2020-12-05T19:05:21.248 INFO:teuthology.orchestra.run.smithi185.stdout: mon reweight min bytes per osd = 10 2020-12-05T19:05:21.248 INFO:teuthology.orchestra.run.smithi185.stdout: mon osd crush smoke test = False 2020-12-05T19:05:21.249 INFO:teuthology.orchestra.run.smithi185.stdout: 2020-12-05T19:05:21.249 INFO:teuthology.orchestra.run.smithi185.stdout:[client] 2020-12-05T19:05:21.249 INFO:teuthology.orchestra.run.smithi185.stdout: rgw cache enabled = true 2020-12-05T19:05:21.249 INFO:teuthology.orchestra.run.smithi185.stdout: rgw enable ops log = true 2020-12-05T19:05:21.249 INFO:teuthology.orchestra.run.smithi185.stdout: rgw enable usage log = true 2020-12-05T19:05:21.250 INFO:teuthology.orchestra.run.smithi185.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-12-05T19:05:21.250 INFO:teuthology.orchestra.run.smithi185.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-12-05T19:05:21.250 INFO:teuthology.orchestra.run.smithi185.stdout: debug rgw = 20 2020-12-05T19:05:21.250 INFO:teuthology.orchestra.run.smithi185.stdout: rgw crypt require ssl = False 2020-12-05T19:05:21.251 INFO:teuthology.orchestra.run.smithi185.stdout: rgw crypt s3 kms backend = testing 2020-12-05T19:05:21.251 INFO:teuthology.orchestra.run.smithi185.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-12-05T19:05:21.251 INFO:teuthology.orchestra.run.smithi185.stdout: rgw curl low speed time = 300 2020-12-05T19:05:21.251 INFO:teuthology.orchestra.run.smithi185.stdout: rgw data log num shards = 4 2020-12-05T19:05:21.251 INFO:teuthology.orchestra.run.smithi185.stdout: rgw md log max shards = 4 2020-12-05T19:05:21.252 INFO:teuthology.orchestra.run.smithi185.stdout: rgw sync log trim interval = 0 2020-12-05T19:05:21.252 INFO:teuthology.orchestra.run.smithi185.stdout: setgroup = ceph 2020-12-05T19:05:21.252 INFO:teuthology.orchestra.run.smithi185.stdout: setuser = ceph 2020-12-05T19:05:21.252 INFO:teuthology.orchestra.run.smithi185.stdout:[mon.a] 2020-12-05T19:05:21.252 INFO:teuthology.orchestra.run.smithi185.stdout:[osd.0] 2020-12-05T19:05:21.253 INFO:teuthology.orchestra.run.smithi185.stdout: osd_max_omap_entries_per_request = 10 2020-12-05T19:05:21.253 INFO:teuthology.orchestra.run.smithi185.stdout:[osd.1] 2020-12-05T19:05:21.253 INFO:teuthology.orchestra.run.smithi185.stdout: osd_max_omap_entries_per_request = 1000 2020-12-05T19:05:21.253 INFO:teuthology.orchestra.run.smithi185.stdout:[osd.2] 2020-12-05T19:05:21.253 INFO:teuthology.orchestra.run.smithi185.stdout: osd_max_omap_entries_per_request = 10000 2020-12-05T19:05:21.254 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-12-05T19:05:21.254 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-12-05T19:05:21.317 INFO:tasks.ceph:Copying monmap to all nodes... 2020-12-05T19:05:21.326 DEBUG:teuthology.orchestra.remote:smithi023:/etc/ceph/c2.keyring is 206B 2020-12-05T19:05:21.385 DEBUG:teuthology.orchestra.remote:smithi023:/home/ubuntu/cephtest/c2.monmap is 179B 2020-12-05T19:05:21.396 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi185.front.sepia.ceph.com 2020-12-05T19:05:21.397 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:05:21.397 INFO:teuthology.orchestra.run.smithi185:> sudo dd of=/etc/ceph/c2.keyring 2020-12-05T19:05:21.398 INFO:teuthology.orchestra.run.smithi185:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-12-05T19:05:21.403 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-12-05T19:05:21.413 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records in 2020-12-05T19:05:21.413 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records out 2020-12-05T19:05:21.414 INFO:teuthology.orchestra.run.smithi185.stderr:206 bytes copied, 6.605e-05 s, 3.1 MB/s 2020-12-05T19:05:21.414 INFO:teuthology.orchestra.run.smithi185.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-12-05T19:05:21.425 INFO:teuthology.orchestra.run.smithi185:> set -ex 2020-12-05T19:05:21.425 INFO:teuthology.orchestra.run.smithi185:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:21.471 INFO:teuthology.orchestra.run.smithi185.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:21.472 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records in 2020-12-05T19:05:21.473 INFO:teuthology.orchestra.run.smithi185.stderr:0+1 records out 2020-12-05T19:05:21.473 INFO:teuthology.orchestra.run.smithi185.stderr:179 bytes copied, 6.5994e-05 s, 2.7 MB/s 2020-12-05T19:05:21.473 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:05:21.474 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:05:21.474 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/c2.keyring 2020-12-05T19:05:21.474 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-12-05T19:05:21.479 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-12-05T19:05:21.489 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-12-05T19:05:21.489 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-12-05T19:05:21.490 INFO:teuthology.orchestra.run.smithi023.stderr:206 bytes copied, 0.00013778 s, 1.5 MB/s 2020-12-05T19:05:21.490 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-12-05T19:05:21.500 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-12-05T19:05:21.501 INFO:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:21.548 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:21.549 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-12-05T19:05:21.550 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-12-05T19:05:21.550 INFO:teuthology.orchestra.run.smithi023.stderr:179 bytes copied, 0.000131021 s, 1.4 MB/s 2020-12-05T19:05:21.550 INFO:tasks.ceph:Setting up mon nodes... 2020-12-05T19:05:21.550 INFO:tasks.ceph:Setting up mgr nodes... 2020-12-05T19:05:21.551 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-12-05T19:05:21.634 INFO:teuthology.orchestra.run.smithi023.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-12-05T19:05:21.636 INFO:tasks.ceph:Setting up mds nodes... 2020-12-05T19:05:21.636 INFO:tasks.ceph_client:Setting up client nodes... 2020-12-05T19:05:21.637 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-12-05T19:05:21.715 INFO:teuthology.orchestra.run.smithi023.stdout:creating /etc/ceph/c2.client.0.keyring 2020-12-05T19:05:21.727 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-12-05T19:05:21.800 INFO:teuthology.orchestra.run.smithi023.stdout:creating /etc/ceph/c2.client.1.keyring 2020-12-05T19:05:21.813 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-12-05T19:05:21.814 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi185.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi023.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-12-05T19:05:21.814 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-12-05T19:05:21.869 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-12-05T19:05:21.869 INFO:tasks.ceph:role: c2.osd.0 2020-12-05T19:05:21.870 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:05:21.870 INFO:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-12-05T19:05:23.258 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-12-05T19:05:23.258 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2020-12-05T19:05:23.258 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-12-05T19:05:23.259 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-12-05T19:05:23.259 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2020-12-05T19:05:23.259 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-12-05T19:05:23.259 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-12-05T19:05:23.260 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-12-05T19:05:23.260 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-12-05T19:05:23.261 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2020-12-05T19:05:23.261 INFO:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-12-05T19:05:23.282 INFO:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-12-05T19:05:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: /sbin/restorecon: command not found 2020-12-05T19:05:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-12-05T19:05:23.336 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-12-05T19:05:23.394 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-12-05T19:05:23.395 INFO:tasks.ceph:role: c2.osd.1 2020-12-05T19:05:23.396 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:05:23.396 INFO:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-12-05T19:05:24.786 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-12-05T19:05:24.786 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2020-12-05T19:05:24.787 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-12-05T19:05:24.787 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-12-05T19:05:24.787 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2020-12-05T19:05:24.787 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-12-05T19:05:24.787 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-12-05T19:05:24.788 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-12-05T19:05:24.788 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-12-05T19:05:24.789 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2020-12-05T19:05:24.789 INFO:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-12-05T19:05:24.810 INFO:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-12-05T19:05:24.862 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: /sbin/restorecon: command not found 2020-12-05T19:05:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-12-05T19:05:24.863 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-12-05T19:05:24.918 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-12-05T19:05:24.919 INFO:tasks.ceph:role: c2.osd.2 2020-12-05T19:05:24.919 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi023.front.sepia.ceph.com 2020-12-05T19:05:24.920 INFO:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-12-05T19:05:26.300 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-12-05T19:05:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2020-12-05T19:05:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-12-05T19:05:26.301 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-12-05T19:05:26.301 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2020-12-05T19:05:26.302 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-12-05T19:05:26.302 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-12-05T19:05:26.302 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-12-05T19:05:26.302 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-12-05T19:05:26.303 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2020-12-05T19:05:26.304 INFO:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-12-05T19:05:26.325 INFO:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-12-05T19:05:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: /sbin/restorecon: command not found 2020-12-05T19:05:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-12-05T19:05:26.379 INFO:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:26.461+0000 7fa844770d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-12-05T19:05:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:26.461+0000 7fa844770d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-12-05T19:05:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:26.461+0000 7fa844770d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-12-05T19:05:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:26.461+0000 7fa844770d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-12-05T19:05:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:26.461+0000 7fa844770d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-12-05T19:05:26.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:26.461+0000 7fa844770d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-12-05T19:05:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:26.485+0000 7fa844770d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-12-05T19:05:28.391 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-12-05T19:05:28.447 INFO:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:28.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:28.525+0000 7f5bf1be2d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-12-05T19:05:28.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:28.525+0000 7f5bf1be2d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-12-05T19:05:28.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:28.525+0000 7f5bf1be2d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-12-05T19:05:28.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:28.525+0000 7f5bf1be2d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-12-05T19:05:28.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:28.525+0000 7f5bf1be2d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-12-05T19:05:28.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:28.529+0000 7f5bf1be2d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-12-05T19:05:28.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:28.557+0000 7f5bf1be2d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-12-05T19:05:30.463 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-12-05T19:05:30.519 INFO:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:30.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:30.601+0000 7f29021d3d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-12-05T19:05:30.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:30.601+0000 7f29021d3d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-12-05T19:05:30.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:30.601+0000 7f29021d3d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-12-05T19:05:30.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:30.601+0000 7f29021d3d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-12-05T19:05:30.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:30.601+0000 7f29021d3d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-12-05T19:05:30.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:30.601+0000 7f29021d3d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-12-05T19:05:30.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:05:30.641+0000 7f29021d3d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-12-05T19:05:32.526 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-12-05T19:05:32.583 INFO:tasks.ceph:Reading keys from all nodes... 2020-12-05T19:05:32.584 INFO:teuthology.orchestra.run.smithi023:> mktemp 2020-12-05T19:05:32.629 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.e5l9ltBNyt 2020-12-05T19:05:32.629 INFO:teuthology.orchestra.run.smithi023:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.e5l9ltBNyt 2020-12-05T19:05:32.686 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.e5l9ltBNyt 2020-12-05T19:05:32.791 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.e5l9ltBNyt is 56B 2020-12-05T19:05:32.803 INFO:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.e5l9ltBNyt 2020-12-05T19:05:32.809 INFO:teuthology.orchestra.run.smithi023:> mktemp 2020-12-05T19:05:32.857 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.EBoRLqu15p 2020-12-05T19:05:32.857 INFO:teuthology.orchestra.run.smithi023:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.EBoRLqu15p 2020-12-05T19:05:32.915 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.EBoRLqu15p 2020-12-05T19:05:33.019 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.EBoRLqu15p is 56B 2020-12-05T19:05:33.030 INFO:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.EBoRLqu15p 2020-12-05T19:05:33.036 INFO:teuthology.orchestra.run.smithi023:> mktemp 2020-12-05T19:05:33.085 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.GDHxsyHK7x 2020-12-05T19:05:33.085 INFO:teuthology.orchestra.run.smithi023:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.GDHxsyHK7x 2020-12-05T19:05:33.142 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.GDHxsyHK7x 2020-12-05T19:05:33.247 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.GDHxsyHK7x is 56B 2020-12-05T19:05:33.258 INFO:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.GDHxsyHK7x 2020-12-05T19:05:33.265 INFO:teuthology.orchestra.run.smithi023:> mktemp 2020-12-05T19:05:33.313 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.VuQXl4wvjT 2020-12-05T19:05:33.313 INFO:teuthology.orchestra.run.smithi023:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.VuQXl4wvjT 2020-12-05T19:05:33.370 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.VuQXl4wvjT 2020-12-05T19:05:33.475 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.VuQXl4wvjT is 56B 2020-12-05T19:05:33.487 INFO:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.VuQXl4wvjT 2020-12-05T19:05:33.543 DEBUG:teuthology.orchestra.remote:smithi023:/etc/ceph/c2.client.0.keyring is 59B 2020-12-05T19:05:33.562 DEBUG:teuthology.orchestra.remote:smithi023:/etc/ceph/c2.client.1.keyring is 59B 2020-12-05T19:05:33.571 INFO:tasks.ceph:Adding keys to all mons... 2020-12-05T19:05:33.572 INFO:teuthology.orchestra.run.smithi023:> sudo tee -a /etc/ceph/c2.keyring 2020-12-05T19:05:33.584 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr.x] 2020-12-05T19:05:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQDx2ctffDIHJhAAZTDFch04sDRp6GLWWWXj6Q== 2020-12-05T19:05:33.584 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.0] 2020-12-05T19:05:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQD22ctfhWO4GxAAUBu0ry3Q69umXC8cKj3gEA== 2020-12-05T19:05:33.585 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.1] 2020-12-05T19:05:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQD42ctfO8uYHxAAMyjDAvwCQEdEpllg+VKLwA== 2020-12-05T19:05:33.585 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.2] 2020-12-05T19:05:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQD62ctf0NERJBAAnt/JIRWXwUNzkTavw6BQUQ== 2020-12-05T19:05:33.586 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2020-12-05T19:05:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQDx2ctf31fZKhAA5YQjplY6hMm08EE9ioBIUw== 2020-12-05T19:05:33.586 INFO:teuthology.orchestra.run.smithi023.stdout:[client.1] 2020-12-05T19:05:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQDx2ctfRmLxLxAA7iTupS6n5ZJoMbHsyHHsrw== 2020-12-05T19:05:33.587 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-12-05T19:05:33.673 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-12-05T19:05:33.757 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-12-05T19:05:33.841 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-12-05T19:05:33.925 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-12-05T19:05:34.014 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-12-05T19:05:34.058 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-12-05T19:05:34.058 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-12-05T19:05:34.114 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-12-05T19:05:34.228 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-12-05T19:05:34.242 INFO:teuthology.orchestra.run.smithi023:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-12-05T19:05:34.289 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-12-05T19:05:34.290 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-12-05T19:05:34.290 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-12-05T19:05:34.333 INFO:tasks.ceph.c2.mon.a:Started 2020-12-05T19:05:34.333 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-12-05T19:05:34.334 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-12-05T19:05:34.334 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-12-05T19:05:34.337 INFO:tasks.ceph.c2.mgr.x:Started 2020-12-05T19:05:34.337 INFO:tasks.ceph:Setting crush tunables to default 2020-12-05T19:05:34.338 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster c2 osd crush tunables default 2020-12-05T19:05:34.669 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2020-12-05T19:05:34.679 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-12-05T19:05:34.680 INFO:teuthology.orchestra.run.smithi023:> mktemp 2020-12-05T19:05:34.684 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.4eGkBeeOEH 2020-12-05T19:05:34.685 INFO:teuthology.orchestra.run.smithi023:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.4eGkBeeOEH 2020-12-05T19:05:34.742 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.4eGkBeeOEH 2020-12-05T19:05:34.847 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.4eGkBeeOEH is 37B 2020-12-05T19:05:34.857 INFO:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.4eGkBeeOEH 2020-12-05T19:05:34.863 INFO:teuthology.orchestra.run.smithi023:> mktemp 2020-12-05T19:05:34.913 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.sqgAceziCI 2020-12-05T19:05:34.914 INFO:teuthology.orchestra.run.smithi023:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.sqgAceziCI 2020-12-05T19:05:34.970 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.sqgAceziCI 2020-12-05T19:05:35.074 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.sqgAceziCI is 37B 2020-12-05T19:05:35.084 INFO:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.sqgAceziCI 2020-12-05T19:05:35.090 INFO:teuthology.orchestra.run.smithi023:> mktemp 2020-12-05T19:05:35.137 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.koAhZnCxwl 2020-12-05T19:05:35.137 INFO:teuthology.orchestra.run.smithi023:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.koAhZnCxwl 2020-12-05T19:05:35.194 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.koAhZnCxwl 2020-12-05T19:05:35.299 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.koAhZnCxwl is 37B 2020-12-05T19:05:35.309 INFO:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.koAhZnCxwl 2020-12-05T19:05:35.315 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster c2 osd new 0f944f1e-5f33-418d-b5a8-af9f9001fc89 0 2020-12-05T19:05:35.566 INFO:teuthology.orchestra.run.smithi023.stdout:0 2020-12-05T19:05:35.580 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster c2 osd new 174c0c1b-4f50-4de2-a3dd-5dc8a222e446 1 2020-12-05T19:05:35.789 INFO:teuthology.orchestra.run.smithi023.stdout:1 2020-12-05T19:05:35.803 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster c2 osd new e038f263-afdb-4a5e-9be1-b45296ae5a12 2 2020-12-05T19:05:35.995 INFO:teuthology.orchestra.run.smithi023.stdout:2 2020-12-05T19:05:36.008 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-12-05T19:05:36.008 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-12-05T19:05:36.010 INFO:tasks.ceph.c2.osd.0:Started 2020-12-05T19:05:36.011 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-12-05T19:05:36.011 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-12-05T19:05:36.013 INFO:tasks.ceph.c2.osd.1:Started 2020-12-05T19:05:36.014 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-12-05T19:05:36.014 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-12-05T19:05:36.016 INFO:tasks.ceph.c2.osd.2:Started 2020-12-05T19:05:36.017 INFO:tasks.ceph:Waiting for OSDs to come up 2020-12-05T19:05:36.321 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-12-05T19:05:36.503 INFO:teuthology.misc.health.smithi023.stdout: 2020-12-05T19:05:36.503 INFO:teuthology.misc.health.smithi023.stdout:{"epoch":5,"fsid":"5bc95e66-3ac9-4ec0-832b-f0600f6214f8","created":"2020-12-05T19:05:34.481496+0000","modified":"2020-12-05T19:05:35.995627+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"0f944f1e-5f33-418d-b5a8-af9f9001fc89","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"174c0c1b-4f50-4de2-a3dd-5dc8a222e446","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e038f263-afdb-4a5e-9be1-b45296ae5a12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:36.513 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-12-05T19:05:36.626 INFO:tasks.ceph.c2.osd.1.smithi023.stderr:2020-12-05T19:05:36.625+0000 7fb1c57bad80 -1 Falling back to public interface 2020-12-05T19:05:36.626 INFO:tasks.ceph.c2.osd.0.smithi023.stderr:2020-12-05T19:05:36.625+0000 7f5c9884bd80 -1 Falling back to public interface 2020-12-05T19:05:36.627 INFO:tasks.ceph.c2.osd.2.smithi023.stderr:2020-12-05T19:05:36.625+0000 7f5e3c7f0d80 -1 Falling back to public interface 2020-12-05T19:05:37.009 INFO:tasks.ceph.c2.osd.1.smithi023.stderr:2020-12-05T19:05:37.009+0000 7fb1c57bad80 -1 osd.1 0 log_to_monitors {default=true} 2020-12-05T19:05:37.259 INFO:tasks.ceph.c2.osd.0.smithi023.stderr:2020-12-05T19:05:37.257+0000 7f5c9884bd80 -1 osd.0 0 log_to_monitors {default=true} 2020-12-05T19:05:37.269 INFO:tasks.ceph.c2.osd.2.smithi023.stderr:2020-12-05T19:05:37.269+0000 7f5e3c7f0d80 -1 osd.2 0 log_to_monitors {default=true} 2020-12-05T19:05:38.517 INFO:tasks.ceph.c2.osd.2.smithi023.stderr:2020-12-05T19:05:38.517+0000 7f5e2623c700 -1 osd.2 0 waiting for initial osdmap 2020-12-05T19:05:38.517 INFO:tasks.ceph.c2.osd.0.smithi023.stderr:2020-12-05T19:05:38.517+0000 7f5c82297700 -1 osd.0 0 waiting for initial osdmap 2020-12-05T19:05:38.518 INFO:tasks.ceph.c2.osd.1.smithi023.stderr:2020-12-05T19:05:38.517+0000 7fb1af206700 -1 osd.1 0 waiting for initial osdmap 2020-12-05T19:05:42.817 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-12-05T19:05:43.126 INFO:teuthology.misc.health.smithi023.stdout: 2020-12-05T19:05:43.126 INFO:teuthology.misc.health.smithi023.stdout:{"epoch":10,"fsid":"5bc95e66-3ac9-4ec0-832b-f0600f6214f8","created":"2020-12-05T19:05:34.481496+0000","modified":"2020-12-05T19:05:41.565825+0000","last_up_change":"2020-12-05T19:05:39.517863+0000","last_in_change":"2020-12-05T19:05:39.517863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-05T19:05:38.563669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"0f944f1e-5f33-418d-b5a8-af9f9001fc89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":10266},{"type":"v1","addr":"172.21.15.23:6808","nonce":10266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6811","nonce":10266},{"type":"v1","addr":"172.21.15.23:6813","nonce":10266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":10266},{"type":"v1","addr":"172.21.15.23:6822","nonce":10266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":10266},{"type":"v1","addr":"172.21.15.23:6818","nonce":10266}]},"public_addr":"172.21.15.23:6808/10266","cluster_addr":"172.21.15.23:6813/10266","heartbeat_back_addr":"172.21.15.23:6822/10266","heartbeat_front_addr":"172.21.15.23:6818/10266","state":["exists","up"]},{"osd":1,"uuid":"174c0c1b-4f50-4de2-a3dd-5dc8a222e446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":10268},{"type":"v1","addr":"172.21.15.23:6801","nonce":10268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":10268},{"type":"v1","addr":"172.21.15.23:6803","nonce":10268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":10268},{"type":"v1","addr":"172.21.15.23:6814","nonce":10268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6805","nonce":10268},{"type":"v1","addr":"172.21.15.23:6807","nonce":10268}]},"public_addr":"172.21.15.23:6801/10268","cluster_addr":"172.21.15.23:6803/10268","heartbeat_back_addr":"172.21.15.23:6814/10268","heartbeat_front_addr":"172.21.15.23:6807/10268","state":["exists","up"]},{"osd":2,"uuid":"e038f263-afdb-4a5e-9be1-b45296ae5a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":10267},{"type":"v1","addr":"172.21.15.23:6809","nonce":10267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":10267},{"type":"v1","addr":"172.21.15.23:6815","nonce":10267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6821","nonce":10267},{"type":"v1","addr":"172.21.15.23:6823","nonce":10267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6817","nonce":10267},{"type":"v1","addr":"172.21.15.23:6819","nonce":10267}]},"public_addr":"172.21.15.23:6809/10267","cluster_addr":"172.21.15.23:6815/10267","heartbeat_back_addr":"172.21.15.23:6823/10267","heartbeat_front_addr":"172.21.15.23:6819/10267","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.288976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.063126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.320850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:43.138 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-12-05T19:05:43.139 INFO:tasks.ceph:Creating RBD pool 2020-12-05T19:05:43.139 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster c2 osd pool create rbd 8 2020-12-05T19:05:43.599 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'rbd' created 2020-12-05T19:05:43.614 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-12-05T19:05:44.610 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rbd' on pool 'rbd' 2020-12-05T19:05:44.623 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-12-05T19:05:44.623 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-12-05T19:05:44.624 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-12-05T19:05:44.887 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:44.887 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":12,"fsid":"5bc95e66-3ac9-4ec0-832b-f0600f6214f8","created":"2020-12-05T19:05:34.481496+0000","modified":"2020-12-05T19:05:44.593176+0000","last_up_change":"2020-12-05T19:05:39.517863+0000","last_in_change":"2020-12-05T19:05:39.517863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-05T19:05:38.563669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-12-05T19:05:43.413413+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0f944f1e-5f33-418d-b5a8-af9f9001fc89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":10266},{"type":"v1","addr":"172.21.15.23:6808","nonce":10266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6811","nonce":10266},{"type":"v1","addr":"172.21.15.23:6813","nonce":10266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":10266},{"type":"v1","addr":"172.21.15.23:6822","nonce":10266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":10266},{"type":"v1","addr":"172.21.15.23:6818","nonce":10266}]},"public_addr":"172.21.15.23:6808/10266","cluster_addr":"172.21.15.23:6813/10266","heartbeat_back_addr":"172.21.15.23:6822/10266","heartbeat_front_addr":"172.21.15.23:6818/10266","state":["exists","up"]},{"osd":1,"uuid":"174c0c1b-4f50-4de2-a3dd-5dc8a222e446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":10268},{"type":"v1","addr":"172.21.15.23:6801","nonce":10268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":10268},{"type":"v1","addr":"172.21.15.23:6803","nonce":10268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":10268},{"type":"v1","addr":"172.21.15.23:6814","nonce":10268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6805","nonce":10268},{"type":"v1","addr":"172.21.15.23:6807","nonce":10268}]},"public_addr":"172.21.15.23:6801/10268","cluster_addr":"172.21.15.23:6803/10268","heartbeat_back_addr":"172.21.15.23:6814/10268","heartbeat_front_addr":"172.21.15.23:6807/10268","state":["exists","up"]},{"osd":2,"uuid":"e038f263-afdb-4a5e-9be1-b45296ae5a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":10267},{"type":"v1","addr":"172.21.15.23:6809","nonce":10267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":10267},{"type":"v1","addr":"172.21.15.23:6815","nonce":10267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6821","nonce":10267},{"type":"v1","addr":"172.21.15.23:6823","nonce":10267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6817","nonce":10267},{"type":"v1","addr":"172.21.15.23:6819","nonce":10267}]},"public_addr":"172.21.15.23:6809/10267","cluster_addr":"172.21.15.23:6815/10267","heartbeat_back_addr":"172.21.15.23:6823/10267","heartbeat_front_addr":"172.21.15.23:6819/10267","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.288976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.063126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.320850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:44.900 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-12-05T19:05:38.563669+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-12-05T19:05:43.413413+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-12-05T19:05:44.901 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2020-12-05T19:05:45.177 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2020-12-05T19:05:45.190 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2020-12-05T19:05:45.470 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-12-05T19:05:45.484 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-12-05T19:05:45.484 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-12-05T19:05:45.485 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2020-12-05T19:05:45.789 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:45.802 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":10057},{"type":"v1","addr":"172.21.15.23:6825","nonce":10057}]},"active_addr":"172.21.15.23:6825/10057","active_change":"2020-12-05T19:05:38.537611+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":2376281260}]},{"addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":3148251377}]},{"addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":3392642386}]}]}} 2020-12-05T19:05:45.804 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-12-05T19:05:45.804 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-12-05T19:05:45.805 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-12-05T19:05:46.075 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:46.075 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":13,"fsid":"5bc95e66-3ac9-4ec0-832b-f0600f6214f8","created":"2020-12-05T19:05:34.481496+0000","modified":"2020-12-05T19:05:45.611703+0000","last_up_change":"2020-12-05T19:05:39.517863+0000","last_in_change":"2020-12-05T19:05:39.517863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-05T19:05:38.563669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-12-05T19:05:43.413413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0f944f1e-5f33-418d-b5a8-af9f9001fc89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":10266},{"type":"v1","addr":"172.21.15.23:6808","nonce":10266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6811","nonce":10266},{"type":"v1","addr":"172.21.15.23:6813","nonce":10266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":10266},{"type":"v1","addr":"172.21.15.23:6822","nonce":10266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":10266},{"type":"v1","addr":"172.21.15.23:6818","nonce":10266}]},"public_addr":"172.21.15.23:6808/10266","cluster_addr":"172.21.15.23:6813/10266","heartbeat_back_addr":"172.21.15.23:6822/10266","heartbeat_front_addr":"172.21.15.23:6818/10266","state":["exists","up"]},{"osd":1,"uuid":"174c0c1b-4f50-4de2-a3dd-5dc8a222e446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":10268},{"type":"v1","addr":"172.21.15.23:6801","nonce":10268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":10268},{"type":"v1","addr":"172.21.15.23:6803","nonce":10268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":10268},{"type":"v1","addr":"172.21.15.23:6814","nonce":10268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6805","nonce":10268},{"type":"v1","addr":"172.21.15.23:6807","nonce":10268}]},"public_addr":"172.21.15.23:6801/10268","cluster_addr":"172.21.15.23:6803/10268","heartbeat_back_addr":"172.21.15.23:6814/10268","heartbeat_front_addr":"172.21.15.23:6807/10268","state":["exists","up"]},{"osd":2,"uuid":"e038f263-afdb-4a5e-9be1-b45296ae5a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":10267},{"type":"v1","addr":"172.21.15.23:6809","nonce":10267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":10267},{"type":"v1","addr":"172.21.15.23:6815","nonce":10267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6821","nonce":10267},{"type":"v1","addr":"172.21.15.23:6823","nonce":10267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6817","nonce":10267},{"type":"v1","addr":"172.21.15.23:6819","nonce":10267}]},"public_addr":"172.21.15.23:6809/10267","cluster_addr":"172.21.15.23:6815/10267","heartbeat_back_addr":"172.21.15.23:6823/10267","heartbeat_front_addr":"172.21.15.23:6819/10267","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.288976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.063126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.320850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:46.088 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-12-05T19:05:46.089 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-12-05T19:05:46.358 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:46.358 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":13,"fsid":"5bc95e66-3ac9-4ec0-832b-f0600f6214f8","created":"2020-12-05T19:05:34.481496+0000","modified":"2020-12-05T19:05:45.611703+0000","last_up_change":"2020-12-05T19:05:39.517863+0000","last_in_change":"2020-12-05T19:05:39.517863+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-12-05T19:05:38.563669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-12-05T19:05:43.413413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0f944f1e-5f33-418d-b5a8-af9f9001fc89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":10266},{"type":"v1","addr":"172.21.15.23:6808","nonce":10266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6811","nonce":10266},{"type":"v1","addr":"172.21.15.23:6813","nonce":10266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":10266},{"type":"v1","addr":"172.21.15.23:6822","nonce":10266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":10266},{"type":"v1","addr":"172.21.15.23:6818","nonce":10266}]},"public_addr":"172.21.15.23:6808/10266","cluster_addr":"172.21.15.23:6813/10266","heartbeat_back_addr":"172.21.15.23:6822/10266","heartbeat_front_addr":"172.21.15.23:6818/10266","state":["exists","up"]},{"osd":1,"uuid":"174c0c1b-4f50-4de2-a3dd-5dc8a222e446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":10268},{"type":"v1","addr":"172.21.15.23:6801","nonce":10268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":10268},{"type":"v1","addr":"172.21.15.23:6803","nonce":10268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":10268},{"type":"v1","addr":"172.21.15.23:6814","nonce":10268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6805","nonce":10268},{"type":"v1","addr":"172.21.15.23:6807","nonce":10268}]},"public_addr":"172.21.15.23:6801/10268","cluster_addr":"172.21.15.23:6803/10268","heartbeat_back_addr":"172.21.15.23:6814/10268","heartbeat_front_addr":"172.21.15.23:6807/10268","state":["exists","up"]},{"osd":2,"uuid":"e038f263-afdb-4a5e-9be1-b45296ae5a12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":10267},{"type":"v1","addr":"172.21.15.23:6809","nonce":10267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":10267},{"type":"v1","addr":"172.21.15.23:6815","nonce":10267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6821","nonce":10267},{"type":"v1","addr":"172.21.15.23:6823","nonce":10267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6817","nonce":10267},{"type":"v1","addr":"172.21.15.23:6819","nonce":10267}]},"public_addr":"172.21.15.23:6809/10267","cluster_addr":"172.21.15.23:6815/10267","heartbeat_back_addr":"172.21.15.23:6823/10267","heartbeat_front_addr":"172.21.15.23:6819/10267","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.288976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.063126+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2020-12-05T19:05:38.320850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-12-05T19:05:46.373 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2020-12-05T19:05:46.531 INFO:teuthology.orchestra.run.smithi023.stdout:34359738371 2020-12-05T19:05:46.532 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2020-12-05T19:05:46.691 INFO:teuthology.orchestra.run.smithi023.stdout:34359738371 2020-12-05T19:05:46.692 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2020-12-05T19:05:46.858 INFO:teuthology.orchestra.run.smithi023.stdout:34359738371 2020-12-05T19:05:46.858 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-12-05T19:05:47.138 INFO:teuthology.orchestra.run.smithi023.stdout:34359738371 2020-12-05T19:05:47.151 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2020-12-05T19:05:47.151 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-12-05T19:05:47.418 INFO:teuthology.orchestra.run.smithi023.stdout:34359738370 2020-12-05T19:05:47.431 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2020-12-05T19:05:48.432 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-12-05T19:05:48.715 INFO:teuthology.orchestra.run.smithi023.stdout:34359738371 2020-12-05T19:05:48.728 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2020-12-05T19:05:48.728 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2020-12-05T19:05:48.993 INFO:teuthology.orchestra.run.smithi023.stdout:34359738371 2020-12-05T19:05:49.006 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2020-12-05T19:05:49.007 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-12-05T19:05:49.007 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-12-05T19:05:49.277 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:49.279 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-12-05T19:05:49.293 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-12-05T19:05:48.545040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146088,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426712,"statfs":{"total":322122547200,"available":318900953088,"internally_reserved":3221225472,"allocated":344064,"data_stored":98208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.940512"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.624692+0000","last_change":"2020-12-05T19:05:44.619122+0000","last_active":"2020-12-05T19:05:44.624692+0000","last_peered":"2020-12-05T19:05:44.624692+0000","last_clean":"2020-12-05T19:05:44.624692+0000","last_became_active":"2020-12-05T19:05:44.618711+0000","last_became_peered":"2020-12-05T19:05:44.618711+0000","last_unstale":"2020-12-05T19:05:44.624692+0000","last_undegraded":"2020-12-05T19:05:44.624692+0000","last_fullsized":"2020-12-05T19:05:44.624692+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.622201+0000","last_change":"2020-12-05T19:05:44.616193+0000","last_active":"2020-12-05T19:05:44.622201+0000","last_peered":"2020-12-05T19:05:44.622201+0000","last_clean":"2020-12-05T19:05:44.622201+0000","last_became_active":"2020-12-05T19:05:44.615735+0000","last_became_peered":"2020-12-05T19:05:44.615735+0000","last_unstale":"2020-12-05T19:05:44.622201+0000","last_undegraded":"2020-12-05T19:05:44.622201+0000","last_fullsized":"2020-12-05T19:05:44.622201+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.624702+0000","last_change":"2020-12-05T19:05:44.618985+0000","last_active":"2020-12-05T19:05:44.624702+0000","last_peered":"2020-12-05T19:05:44.624702+0000","last_clean":"2020-12-05T19:05:44.624702+0000","last_became_active":"2020-12-05T19:05:44.618560+0000","last_became_peered":"2020-12-05T19:05:44.618560+0000","last_unstale":"2020-12-05T19:05:44.624702+0000","last_undegraded":"2020-12-05T19:05:44.624702+0000","last_fullsized":"2020-12-05T19:05:44.624702+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.614957+0000","last_change":"2020-12-05T19:05:44.612800+0000","last_active":"2020-12-05T19:05:44.614957+0000","last_peered":"2020-12-05T19:05:44.614957+0000","last_clean":"2020-12-05T19:05:44.614957+0000","last_became_active":"2020-12-05T19:05:44.612361+0000","last_became_peered":"2020-12-05T19:05:44.612361+0000","last_unstale":"2020-12-05T19:05:44.614957+0000","last_undegraded":"2020-12-05T19:05:44.614957+0000","last_fullsized":"2020-12-05T19:05:44.614957+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.611503+0000","last_change":"2020-12-05T19:05:41.272353+0000","last_active":"2020-12-05T19:05:44.611503+0000","last_peered":"2020-12-05T19:05:44.611503+0000","last_clean":"2020-12-05T19:05:44.611503+0000","last_became_active":"2020-12-05T19:05:41.271112+0000","last_became_peered":"2020-12-05T19:05:41.271112+0000","last_unstale":"2020-12-05T19:05:44.611503+0000","last_undegraded":"2020-12-05T19:05:44.611503+0000","last_fullsized":"2020-12-05T19:05:44.611503+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:39.517863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:39.517863+0000","last_clean_scrub_stamp":"2020-12-05T19:05:39.517863+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.617847+0000","last_change":"2020-12-05T19:05:44.613933+0000","last_active":"2020-12-05T19:05:44.617847+0000","last_peered":"2020-12-05T19:05:44.617847+0000","last_clean":"2020-12-05T19:05:44.617847+0000","last_became_active":"2020-12-05T19:05:44.613392+0000","last_became_peered":"2020-12-05T19:05:44.613392+0000","last_unstale":"2020-12-05T19:05:44.617847+0000","last_undegraded":"2020-12-05T19:05:44.617847+0000","last_fullsized":"2020-12-05T19:05:44.617847+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.620101+0000","last_change":"2020-12-05T19:05:44.616811+0000","last_active":"2020-12-05T19:05:44.620101+0000","last_peered":"2020-12-05T19:05:44.620101+0000","last_clean":"2020-12-05T19:05:44.620101+0000","last_became_active":"2020-12-05T19:05:44.616266+0000","last_became_peered":"2020-12-05T19:05:44.616266+0000","last_unstale":"2020-12-05T19:05:44.620101+0000","last_undegraded":"2020-12-05T19:05:44.620101+0000","last_fullsized":"2020-12-05T19:05:44.620101+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.624525+0000","last_change":"2020-12-05T19:05:44.618712+0000","last_active":"2020-12-05T19:05:44.624525+0000","last_peered":"2020-12-05T19:05:44.624525+0000","last_clean":"2020-12-05T19:05:44.624525+0000","last_became_active":"2020-12-05T19:05:44.618250+0000","last_became_peered":"2020-12-05T19:05:44.618250+0000","last_unstale":"2020-12-05T19:05:44.624525+0000","last_undegraded":"2020-12-05T19:05:44.624525+0000","last_fullsized":"2020-12-05T19:05:44.624525+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.617504+0000","last_change":"2020-12-05T19:05:44.614384+0000","last_active":"2020-12-05T19:05:44.617504+0000","last_peered":"2020-12-05T19:05:44.617504+0000","last_clean":"2020-12-05T19:05:44.617504+0000","last_became_active":"2020-12-05T19:05:44.613975+0000","last_became_peered":"2020-12-05T19:05:44.613975+0000","last_unstale":"2020-12-05T19:05:44.617504+0000","last_undegraded":"2020-12-05T19:05:44.617504+0000","last_fullsized":"2020-12-05T19:05:44.617504+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.893}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]}]}]}} 2020-12-05T19:05:49.295 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-12-05T19:05:49.557 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:49.560 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-12-05T19:05:49.578 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-12-05T19:05:48.545040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146088,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426712,"statfs":{"total":322122547200,"available":318900953088,"internally_reserved":3221225472,"allocated":344064,"data_stored":98208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.940512"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.624692+0000","last_change":"2020-12-05T19:05:44.619122+0000","last_active":"2020-12-05T19:05:44.624692+0000","last_peered":"2020-12-05T19:05:44.624692+0000","last_clean":"2020-12-05T19:05:44.624692+0000","last_became_active":"2020-12-05T19:05:44.618711+0000","last_became_peered":"2020-12-05T19:05:44.618711+0000","last_unstale":"2020-12-05T19:05:44.624692+0000","last_undegraded":"2020-12-05T19:05:44.624692+0000","last_fullsized":"2020-12-05T19:05:44.624692+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.622201+0000","last_change":"2020-12-05T19:05:44.616193+0000","last_active":"2020-12-05T19:05:44.622201+0000","last_peered":"2020-12-05T19:05:44.622201+0000","last_clean":"2020-12-05T19:05:44.622201+0000","last_became_active":"2020-12-05T19:05:44.615735+0000","last_became_peered":"2020-12-05T19:05:44.615735+0000","last_unstale":"2020-12-05T19:05:44.622201+0000","last_undegraded":"2020-12-05T19:05:44.622201+0000","last_fullsized":"2020-12-05T19:05:44.622201+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.624702+0000","last_change":"2020-12-05T19:05:44.618985+0000","last_active":"2020-12-05T19:05:44.624702+0000","last_peered":"2020-12-05T19:05:44.624702+0000","last_clean":"2020-12-05T19:05:44.624702+0000","last_became_active":"2020-12-05T19:05:44.618560+0000","last_became_peered":"2020-12-05T19:05:44.618560+0000","last_unstale":"2020-12-05T19:05:44.624702+0000","last_undegraded":"2020-12-05T19:05:44.624702+0000","last_fullsized":"2020-12-05T19:05:44.624702+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.614957+0000","last_change":"2020-12-05T19:05:44.612800+0000","last_active":"2020-12-05T19:05:44.614957+0000","last_peered":"2020-12-05T19:05:44.614957+0000","last_clean":"2020-12-05T19:05:44.614957+0000","last_became_active":"2020-12-05T19:05:44.612361+0000","last_became_peered":"2020-12-05T19:05:44.612361+0000","last_unstale":"2020-12-05T19:05:44.614957+0000","last_undegraded":"2020-12-05T19:05:44.614957+0000","last_fullsized":"2020-12-05T19:05:44.614957+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.611503+0000","last_change":"2020-12-05T19:05:41.272353+0000","last_active":"2020-12-05T19:05:44.611503+0000","last_peered":"2020-12-05T19:05:44.611503+0000","last_clean":"2020-12-05T19:05:44.611503+0000","last_became_active":"2020-12-05T19:05:41.271112+0000","last_became_peered":"2020-12-05T19:05:41.271112+0000","last_unstale":"2020-12-05T19:05:44.611503+0000","last_undegraded":"2020-12-05T19:05:44.611503+0000","last_fullsized":"2020-12-05T19:05:44.611503+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:39.517863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:39.517863+0000","last_clean_scrub_stamp":"2020-12-05T19:05:39.517863+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.617847+0000","last_change":"2020-12-05T19:05:44.613933+0000","last_active":"2020-12-05T19:05:44.617847+0000","last_peered":"2020-12-05T19:05:44.617847+0000","last_clean":"2020-12-05T19:05:44.617847+0000","last_became_active":"2020-12-05T19:05:44.613392+0000","last_became_peered":"2020-12-05T19:05:44.613392+0000","last_unstale":"2020-12-05T19:05:44.617847+0000","last_undegraded":"2020-12-05T19:05:44.617847+0000","last_fullsized":"2020-12-05T19:05:44.617847+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.620101+0000","last_change":"2020-12-05T19:05:44.616811+0000","last_active":"2020-12-05T19:05:44.620101+0000","last_peered":"2020-12-05T19:05:44.620101+0000","last_clean":"2020-12-05T19:05:44.620101+0000","last_became_active":"2020-12-05T19:05:44.616266+0000","last_became_peered":"2020-12-05T19:05:44.616266+0000","last_unstale":"2020-12-05T19:05:44.620101+0000","last_undegraded":"2020-12-05T19:05:44.620101+0000","last_fullsized":"2020-12-05T19:05:44.620101+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.624525+0000","last_change":"2020-12-05T19:05:44.618712+0000","last_active":"2020-12-05T19:05:44.624525+0000","last_peered":"2020-12-05T19:05:44.624525+0000","last_clean":"2020-12-05T19:05:44.624525+0000","last_became_active":"2020-12-05T19:05:44.618250+0000","last_became_peered":"2020-12-05T19:05:44.618250+0000","last_unstale":"2020-12-05T19:05:44.624525+0000","last_undegraded":"2020-12-05T19:05:44.624525+0000","last_fullsized":"2020-12-05T19:05:44.624525+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-12-05T19:05:44.617504+0000","last_change":"2020-12-05T19:05:44.614384+0000","last_active":"2020-12-05T19:05:44.617504+0000","last_peered":"2020-12-05T19:05:44.617504+0000","last_clean":"2020-12-05T19:05:44.617504+0000","last_became_active":"2020-12-05T19:05:44.613975+0000","last_became_peered":"2020-12-05T19:05:44.613975+0000","last_unstale":"2020-12-05T19:05:44.617504+0000","last_undegraded":"2020-12-05T19:05:44.617504+0000","last_fullsized":"2020-12-05T19:05:44.617504+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-05T19:05:43.585758+0000","last_clean_scrub_stamp":"2020-12-05T19:05:43.585758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.893}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]}]}]}} 2020-12-05T19:05:49.579 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-12-05T19:05:49.579 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-12-05T19:05:49.580 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-12-05T19:05:49.580 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2020-12-05T19:05:49.897 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-12-05T19:05:49.918 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-12-05T19:05:49.919 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-12-05T19:05:49.919 INFO:teuthology.run_tasks:Running task rgw... 2020-12-05T19:05:49.935 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-12-05T19:05:49.935 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-12-05T19:05:49.936 INFO:tasks.rgw:Creating data pools 2020-12-05T19:05:49.936 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-12-05T19:05:49.936 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-12-05T19:05:50.847 INFO:teuthology.orchestra.run.smithi185.stderr:pool 'default.rgw.buckets.data' created 2020-12-05T19:05:50.861 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-12-05T19:05:51.870 INFO:teuthology.orchestra.run.smithi185.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-12-05T19:05:51.884 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-12-05T19:05:52.856 INFO:teuthology.orchestra.run.smithi185.stderr:pool 'default.rgw.buckets.index' created 2020-12-05T19:05:52.871 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-12-05T19:05:53.892 INFO:teuthology.orchestra.run.smithi185.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-12-05T19:05:53.902 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-12-05T19:05:53.902 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-12-05T19:05:54.151 INFO:teuthology.orchestra.run.smithi185.stderr:pool 'default.rgw.buckets.data' already exists 2020-12-05T19:05:54.161 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-12-05T19:05:54.866 INFO:teuthology.orchestra.run.smithi185.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-12-05T19:05:54.879 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-12-05T19:05:55.153 INFO:teuthology.orchestra.run.smithi185.stderr:pool 'default.rgw.buckets.index' already exists 2020-12-05T19:05:55.163 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-12-05T19:05:55.872 INFO:teuthology.orchestra.run.smithi185.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-12-05T19:05:55.886 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-12-05T19:05:55.886 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-12-05T19:05:56.493 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'default.rgw.buckets.data' created 2020-12-05T19:05:56.514 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-12-05T19:05:57.518 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-12-05T19:05:57.543 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-12-05T19:05:58.520 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'default.rgw.buckets.index' created 2020-12-05T19:05:58.532 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-12-05T19:05:59.540 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-12-05T19:05:59.557 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-12-05T19:05:59.558 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-12-05T19:05:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'default.rgw.buckets.data' already exists 2020-12-05T19:05:59.824 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-12-05T19:06:00.510 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-12-05T19:06:00.523 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-12-05T19:06:00.791 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'default.rgw.buckets.index' already exists 2020-12-05T19:06:00.806 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-12-05T19:06:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-12-05T19:06:01.574 DEBUG:tasks.rgw:Pools created 2020-12-05T19:06:01.574 INFO:tasks.rgw:Configuring compression type = random 2020-12-05T19:06:01.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-12-05T19:06:01.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-12-05T19:06:01.576 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-12-05T19:06:01.612 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:01.612 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:01.621 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:01.622+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.832 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.834+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.832 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.834+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.833 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.834+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.834 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.834+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.836 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.838+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.836 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.838+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.837 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.838+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.854 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.854 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.855 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.855 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.855 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.856 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.856 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 10 cannot find current period zonegroup using local zonegroup 2020-12-05T19:06:02.857 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.857 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.857 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.857 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.858 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 10 Creating default zonegroup 2020-12-05T19:06:02.858 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 20 get_system_obj_state: rctx=0x7ffc7ff08060 obj=:.pools.avail state=0x5615e6ce4750 s->prefetch_data=0 2020-12-05T19:06:02.858 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.858+0000 7f93e7582080 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-12-05T19:06:02.873 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.874+0000 7f93e7582080 10 WARNING: store->list_zones() returned r=-2 2020-12-05T19:06:02.874 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.874+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.874 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.878+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.879 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.882+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.879 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.882+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.880 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.882+0000 7f93e7582080 10 could not read realm id: (2) No such file or directory 2020-12-05T19:06:02.880 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.882+0000 7f93e7582080 10 WARNING: failed to set zone as default, r=-22 2020-12-05T19:06:02.880 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.882+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.881 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.882+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.886 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.887 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 20 rados_obj.operate() r=0 bl.length=839 2020-12-05T19:06:02.888 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.889 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:02.889 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 20 zonegroup default 2020-12-05T19:06:02.889 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.891 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:02.891 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 10 Cannot find current period zone using local zone 2020-12-05T19:06:02.892 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 10 Using default name default 2020-12-05T19:06:02.892 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.890+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.892 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.894+0000 7f93e7582080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:02.893 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.894+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:02.893 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.894+0000 7f93e7582080 20 rados_obj.operate() r=0 bl.length=839 2020-12-05T19:06:02.894 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.894+0000 7f93e7582080 20 zone default found 2020-12-05T19:06:02.894 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:02.894+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:04.843 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:04.846+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:04.843 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:04.846+0000 7f93e7582080 20 rados->read ofs=0 len=0 2020-12-05T19:06:04.851 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:04.854+0000 7f93e7582080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:04.851 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:04.854+0000 7f93e7582080 20 started sync module instance, tier type = 2020-12-05T19:06:04.851 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:04.854+0000 7f93e7582080 20 started zone id=10ae13c9-aabd-4490-92a7-36e09fe35deb (name=default) with tier type = 2020-12-05T19:06:06.905 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.906+0000 7f93e7582080 20 add_watcher() i=0 2020-12-05T19:06:06.906 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.906+0000 7f93e7582080 20 add_watcher() i=1 2020-12-05T19:06:06.906 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.906+0000 7f93e7582080 20 add_watcher() i=2 2020-12-05T19:06:06.907 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.906+0000 7f93e7582080 20 add_watcher() i=3 2020-12-05T19:06:06.907 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.906+0000 7f93e7582080 20 add_watcher() i=4 2020-12-05T19:06:06.907 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.906+0000 7f93e7582080 20 add_watcher() i=5 2020-12-05T19:06:06.908 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.906+0000 7f93e7582080 20 add_watcher() i=6 2020-12-05T19:06:06.909 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.910+0000 7f93e7582080 20 add_watcher() i=7 2020-12-05T19:06:06.909 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.910+0000 7f93e7582080 2 all 8 watchers are set, enabling cache 2020-12-05T19:06:06.910 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.910+0000 7f93e7582080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-12-05T19:06:06.910 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.910+0000 7f93e7582080 20 check_secure_mon_conn(): mode 1 is insecure 2020-12-05T19:06:06.910 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:06.910+0000 7f931ffff700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-12-05T19:06:10.870 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:10.874+0000 7f93e7582080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-12-05T19:06:10.987 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:10.990+0000 7f930e7fc700 20 reqs_thread_entry: start 2020-12-05T19:06:10.988 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:10.990+0000 7f93e7582080 20 init_complete bucket index max shards: 11 2020-12-05T19:06:10.988 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:10.990+0000 7f92fffff700 20 reqs_thread_entry: start 2020-12-05T19:06:10.988 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:10.990+0000 7f92fe7fc700 20 reqs_thread_entry: start 2020-12-05T19:06:11.012 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.014+0000 7f931ffff700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-12-05T19:06:11.015 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.018+0000 7f93e7582080 20 remove_watcher() i=0 2020-12-05T19:06:11.015 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.018+0000 7f93e7582080 2 removed watcher, disabling cache 2020-12-05T19:06:11.017 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.018+0000 7f93e7582080 20 remove_watcher() i=1 2020-12-05T19:06:11.020 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.022+0000 7f93e7582080 20 remove_watcher() i=2 2020-12-05T19:06:11.025 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.026+0000 7f93e7582080 20 remove_watcher() i=3 2020-12-05T19:06:11.028 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.030+0000 7f93e7582080 20 remove_watcher() i=4 2020-12-05T19:06:11.031 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.034+0000 7f93e7582080 20 remove_watcher() i=5 2020-12-05T19:06:11.033 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.034+0000 7f93e7582080 20 remove_watcher() i=6 2020-12-05T19:06:11.036 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.038+0000 7f93e7582080 20 remove_watcher() i=7 2020-12-05T19:06:11.044 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:06:11.045 DEBUG:tasks.util.rgw: json result: [] 2020-12-05T19:06:11.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-12-05T19:06:11.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-12-05T19:06:11.046 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-12-05T19:06:11.125 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:11.125 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:11.134 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.138+0000 7f4ca7998080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-12-05T19:06:11.134 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.138+0000 7f4ca7998080 20 check_secure_mon_conn(): mode 1 is insecure 2020-12-05T19:06:11.139 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.142+0000 7f4bd7fff700 20 reqs_thread_entry: start 2020-12-05T19:06:11.140 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.142+0000 7f4ca7998080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.141 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.142+0000 7f4ca7998080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:11.141 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.142+0000 7f4ca7998080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.142 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.146+0000 7f4ca7998080 20 rados_obj.operate() r=0 bl.length=839 2020-12-05T19:06:11.142 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.146+0000 7f4ca7998080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.142 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.146+0000 7f4ca7998080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.143 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.146+0000 7f4ca7998080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.143 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.146+0000 7f4ca7998080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:11.143 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.146+0000 7f4ca7998080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.144 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.146+0000 7f4ca7998080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:11.154 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"10ae13c9-aabd-4490-92a7-36e09fe35deb","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-12-05T19:06:11.154 DEBUG:tasks.util.rgw: json result: {'id': '10ae13c9-aabd-4490-92a7-36e09fe35deb', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-12-05T19:06:11.155 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-12-05T19:06:11.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-12-05T19:06:11.156 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-12-05T19:06:11.230 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:11.230 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:11.238 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.242+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.239 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.242+0000 7eff73749080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.239 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.242+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.240 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.242+0000 7eff73749080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.240 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.242+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.241 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.242+0000 7eff73749080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.241 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.242+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.241 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.242+0000 7eff73749080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:11.261 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.262+0000 7eff73749080 20 RGWRados::pool_iterate: got zonegroup_info.7da492ce-d215-4da5-beec-1f6f6d0c8005 2020-12-05T19:06:11.261 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.262+0000 7eff73749080 20 RGWRados::pool_iterate: got zone_info.10ae13c9-aabd-4490-92a7-36e09fe35deb 2020-12-05T19:06:11.261 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.262+0000 7eff73749080 20 RGWRados::pool_iterate: got zone_names.default 2020-12-05T19:06:11.262 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.262+0000 7eff73749080 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-12-05T19:06:11.262 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.262+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.262 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.262+0000 7eff73749080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:11.263 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.262+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.265 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.266+0000 7eff73749080 20 rados_obj.operate() r=0 bl.length=855 2020-12-05T19:06:11.265 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.266+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.266 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:11.266 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.267 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:11.267 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.268 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.268 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 10 cannot find current period zonegroup using local zonegroup 2020-12-05T19:06:11.268 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.269 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.269 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.270 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:11.270 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.270 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:11.271 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 zonegroup default 2020-12-05T19:06:11.271 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.270+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.271 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.271 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 10 Cannot find current period zone using local zone 2020-12-05T19:06:11.272 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.273 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.273 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.273 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:11.273 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.274 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados_obj.operate() r=0 bl.length=855 2020-12-05T19:06:11.274 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 zone default found 2020-12-05T19:06:11.274 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.275 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.275 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.274+0000 7eff73749080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.275 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.278+0000 7eff73749080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.276 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.278+0000 7eff73749080 20 started sync module instance, tier type = 2020-12-05T19:06:11.276 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.278+0000 7eff73749080 20 started zone id=10ae13c9-aabd-4490-92a7-36e09fe35deb (name=default) with tier type = 2020-12-05T19:06:11.303 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 add_watcher() i=0 2020-12-05T19:06:11.304 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 add_watcher() i=1 2020-12-05T19:06:11.304 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 add_watcher() i=2 2020-12-05T19:06:11.305 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 add_watcher() i=3 2020-12-05T19:06:11.305 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 add_watcher() i=4 2020-12-05T19:06:11.305 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 add_watcher() i=5 2020-12-05T19:06:11.307 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 add_watcher() i=6 2020-12-05T19:06:11.307 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 add_watcher() i=7 2020-12-05T19:06:11.308 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 2 all 8 watchers are set, enabling cache 2020-12-05T19:06:11.308 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-12-05T19:06:11.308 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff73749080 20 check_secure_mon_conn(): mode 1 is insecure 2020-12-05T19:06:11.309 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.306+0000 7eff2f7ee700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-12-05T19:06:11.398 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.398+0000 7efe9dffb700 20 reqs_thread_entry: start 2020-12-05T19:06:11.398 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.398+0000 7efe977fe700 20 reqs_thread_entry: start 2020-12-05T19:06:11.398 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.402+0000 7eff73749080 20 init_complete bucket index max shards: 11 2020-12-05T19:06:11.399 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.402+0000 7efe95ffb700 20 reqs_thread_entry: start 2020-12-05T19:06:11.424 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.426+0000 7eff2f7ee700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-12-05T19:06:11.427 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.430+0000 7eff73749080 20 remove_watcher() i=0 2020-12-05T19:06:11.427 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.430+0000 7eff73749080 2 removed watcher, disabling cache 2020-12-05T19:06:11.431 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.434+0000 7eff73749080 20 remove_watcher() i=1 2020-12-05T19:06:11.433 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.434+0000 7eff73749080 20 remove_watcher() i=2 2020-12-05T19:06:11.436 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.438+0000 7eff73749080 20 remove_watcher() i=3 2020-12-05T19:06:11.439 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.442+0000 7eff73749080 20 remove_watcher() i=4 2020-12-05T19:06:11.442 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.446+0000 7eff73749080 20 remove_watcher() i=5 2020-12-05T19:06:11.444 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.446+0000 7eff73749080 20 remove_watcher() i=6 2020-12-05T19:06:11.447 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.450+0000 7eff73749080 20 remove_watcher() i=7 2020-12-05T19:06:11.454 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:06:11.455 DEBUG:tasks.util.rgw: json result: [] 2020-12-05T19:06:11.455 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-12-05T19:06:11.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-12-05T19:06:11.456 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-12-05T19:06:11.538 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:11.539 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:11.548 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.549+0000 7f496dd24080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-12-05T19:06:11.549 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.549+0000 7f496dd24080 20 check_secure_mon_conn(): mode 1 is insecure 2020-12-05T19:06:11.556 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.557+0000 7f496dd24080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.557 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.557+0000 7f48a27fc700 20 reqs_thread_entry: start 2020-12-05T19:06:11.558 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.561+0000 7f496dd24080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:11.558 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.561+0000 7f496dd24080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.559 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.561+0000 7f496dd24080 20 rados_obj.operate() r=0 bl.length=855 2020-12-05T19:06:11.559 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.561+0000 7f496dd24080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.559 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.561+0000 7f496dd24080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:11.560 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.561+0000 7f496dd24080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.560 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.561+0000 7f496dd24080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:11.561 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.561+0000 7f496dd24080 20 rados->read ofs=0 len=0 2020-12-05T19:06:11.562 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:11.561+0000 7f496dd24080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:11.569 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"10ae13c9-aabd-4490-92a7-36e09fe35deb","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-12-05T19:06:11.570 DEBUG:tasks.util.rgw: json result: {'id': '10ae13c9-aabd-4490-92a7-36e09fe35deb', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-12-05T19:06:11.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-12-05T19:06:11.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-12-05T19:06:11.571 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-12-05T19:06:11.607 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:11.607 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:11.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:11.617+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.645+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.645+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.649+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.649+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.649+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.649+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.657+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 10 cannot find current period zonegroup using local zonegroup 2020-12-05T19:06:13.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 10 Creating default zonegroup 2020-12-05T19:06:13.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 20 get_system_obj_state: rctx=0x7ffc2b5b13a0 obj=:.pools.avail state=0x5570331ea750 s->prefetch_data=0 2020-12-05T19:06:13.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.677+0000 7fb3e1e38080 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-12-05T19:06:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.693+0000 7fb3e1e38080 10 WARNING: store->list_zones() returned r=-2 2020-12-05T19:06:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.693+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.697+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.701+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.701+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.701+0000 7fb3e1e38080 10 could not read realm id: (2) No such file or directory 2020-12-05T19:06:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.701+0000 7fb3e1e38080 10 WARNING: failed to set zone as default, r=-22 2020-12-05T19:06:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.701+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.701+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.709+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.709+0000 7fb3e1e38080 20 rados_obj.operate() r=0 bl.length=839 2020-12-05T19:06:13.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.709+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.709+0000 7fb3e1e38080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:13.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.709+0000 7fb3e1e38080 20 zonegroup default 2020-12-05T19:06:13.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.709+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.713+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:13.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.713+0000 7fb3e1e38080 10 Cannot find current period zone using local zone 2020-12-05T19:06:13.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.713+0000 7fb3e1e38080 10 Using default name default 2020-12-05T19:06:13.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.713+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.713+0000 7fb3e1e38080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:13.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.713+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:13.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.713+0000 7fb3e1e38080 20 rados_obj.operate() r=0 bl.length=839 2020-12-05T19:06:13.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.713+0000 7fb3e1e38080 20 zone default found 2020-12-05T19:06:13.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:13.713+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:15.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:15.673+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:15.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:15.673+0000 7fb3e1e38080 20 rados->read ofs=0 len=0 2020-12-05T19:06:15.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:15.673+0000 7fb3e1e38080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:15.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:15.673+0000 7fb3e1e38080 20 started sync module instance, tier type = 2020-12-05T19:06:15.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:15.673+0000 7fb3e1e38080 20 started zone id=cb3b981d-613a-41b7-a658-0de320ff75a0 (name=default) with tier type = 2020-12-05T19:06:17.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 add_watcher() i=0 2020-12-05T19:06:17.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 add_watcher() i=1 2020-12-05T19:06:17.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 add_watcher() i=2 2020-12-05T19:06:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 add_watcher() i=3 2020-12-05T19:06:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 add_watcher() i=4 2020-12-05T19:06:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 add_watcher() i=5 2020-12-05T19:06:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 add_watcher() i=6 2020-12-05T19:06:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 add_watcher() i=7 2020-12-05T19:06:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 2 all 8 watchers are set, enabling cache 2020-12-05T19:06:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-12-05T19:06:17.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb3e1e38080 20 check_secure_mon_conn(): mode 1 is insecure 2020-12-05T19:06:17.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:17.729+0000 7fb39efdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-12-05T19:06:21.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.708+0000 7fb3e1e38080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-12-05T19:06:21.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.840+0000 7fb30d7fa700 20 reqs_thread_entry: start 2020-12-05T19:06:21.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.840+0000 7fb3e1e38080 20 init_complete bucket index max shards: 11 2020-12-05T19:06:21.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.840+0000 7fb2faffd700 20 reqs_thread_entry: start 2020-12-05T19:06:21.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.840+0000 7fb2f97fa700 20 reqs_thread_entry: start 2020-12-05T19:06:21.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.864+0000 7fb39efdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-12-05T19:06:21.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.868+0000 7fb3e1e38080 20 remove_watcher() i=0 2020-12-05T19:06:21.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.868+0000 7fb3e1e38080 2 removed watcher, disabling cache 2020-12-05T19:06:21.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.872+0000 7fb3e1e38080 20 remove_watcher() i=1 2020-12-05T19:06:21.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.872+0000 7fb3e1e38080 20 remove_watcher() i=2 2020-12-05T19:06:21.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.876+0000 7fb3e1e38080 20 remove_watcher() i=3 2020-12-05T19:06:21.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.880+0000 7fb3e1e38080 20 remove_watcher() i=4 2020-12-05T19:06:21.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.884+0000 7fb3e1e38080 20 remove_watcher() i=5 2020-12-05T19:06:21.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.884+0000 7fb3e1e38080 20 remove_watcher() i=6 2020-12-05T19:06:21.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.888+0000 7fb3e1e38080 20 remove_watcher() i=7 2020-12-05T19:06:21.898 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2020-12-05T19:06:21.898 DEBUG:tasks.util.rgw: json result: [] 2020-12-05T19:06:21.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-12-05T19:06:21.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-12-05T19:06:21.899 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-12-05T19:06:21.982 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:21.982 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:21.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.992+0000 7fbfeb4b9080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-12-05T19:06:21.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:21.992+0000 7fbfeb4b9080 20 check_secure_mon_conn(): mode 1 is insecure 2020-12-05T19:06:22.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.000+0000 7fbf1effd700 20 reqs_thread_entry: start 2020-12-05T19:06:22.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.000+0000 7fbfeb4b9080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.004+0000 7fbfeb4b9080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:22.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.004+0000 7fbfeb4b9080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.004+0000 7fbfeb4b9080 20 rados_obj.operate() r=0 bl.length=839 2020-12-05T19:06:22.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.004+0000 7fbfeb4b9080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.004+0000 7fbfeb4b9080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.004+0000 7fbfeb4b9080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.004+0000 7fbfeb4b9080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:22.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.004+0000 7fbfeb4b9080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.008+0000 7fbfeb4b9080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:22.019 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"cb3b981d-613a-41b7-a658-0de320ff75a0","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-12-05T19:06:22.019 DEBUG:tasks.util.rgw: json result: {'id': 'cb3b981d-613a-41b7-a658-0de320ff75a0', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-12-05T19:06:22.020 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-12-05T19:06:22.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-12-05T19:06:22.021 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-12-05T19:06:22.093 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:22.093 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:22.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.104+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.104+0000 7f98690e1080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.104+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.104+0000 7f98690e1080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.104+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.104+0000 7f98690e1080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.104+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.108+0000 7f98690e1080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:22.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.132+0000 7f98690e1080 20 RGWRados::pool_iterate: got zonegroup_info.5bc304e3-96c1-4a23-992a-d0d9dd793c91 2020-12-05T19:06:22.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.132+0000 7f98690e1080 20 RGWRados::pool_iterate: got zone_info.cb3b981d-613a-41b7-a658-0de320ff75a0 2020-12-05T19:06:22.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.132+0000 7f98690e1080 20 RGWRados::pool_iterate: got zone_names.default 2020-12-05T19:06:22.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.132+0000 7f98690e1080 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-12-05T19:06:22.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.132+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.132+0000 7f98690e1080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:22.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.132+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.132+0000 7f98690e1080 20 rados_obj.operate() r=0 bl.length=855 2020-12-05T19:06:22.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.132+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:22.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:22.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 10 cannot find current period zonegroup using local zonegroup 2020-12-05T19:06:22.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:22.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.136+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:22.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 zonegroup default 2020-12-05T19:06:22.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 10 Cannot find current period zone using local zone 2020-12-05T19:06:22.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:22.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados_obj.operate() r=0 bl.length=855 2020-12-05T19:06:22.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 zone default found 2020-12-05T19:06:22.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.140+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.144+0000 7f98690e1080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.144+0000 7f98690e1080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.144+0000 7f98690e1080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.144+0000 7f98690e1080 20 started sync module instance, tier type = 2020-12-05T19:06:22.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.144+0000 7f98690e1080 20 started zone id=cb3b981d-613a-41b7-a658-0de320ff75a0 (name=default) with tier type = 2020-12-05T19:06:22.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 add_watcher() i=0 2020-12-05T19:06:22.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 add_watcher() i=1 2020-12-05T19:06:22.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 add_watcher() i=2 2020-12-05T19:06:22.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 add_watcher() i=3 2020-12-05T19:06:22.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 add_watcher() i=4 2020-12-05T19:06:22.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 add_watcher() i=5 2020-12-05T19:06:22.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 add_watcher() i=6 2020-12-05T19:06:22.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 add_watcher() i=7 2020-12-05T19:06:22.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 2 all 8 watchers are set, enabling cache 2020-12-05T19:06:22.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-12-05T19:06:22.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f98690e1080 20 check_secure_mon_conn(): mode 1 is insecure 2020-12-05T19:06:22.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.168+0000 7f9824fe9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-12-05T19:06:22.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.264+0000 7f9796ffd700 20 reqs_thread_entry: start 2020-12-05T19:06:22.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.264+0000 7f9794ff9700 20 reqs_thread_entry: start 2020-12-05T19:06:22.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.264+0000 7f98690e1080 20 init_complete bucket index max shards: 11 2020-12-05T19:06:22.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.268+0000 7f978affd700 20 reqs_thread_entry: start 2020-12-05T19:06:22.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.292+0000 7f9824fe9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-12-05T19:06:22.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.296+0000 7f98690e1080 20 remove_watcher() i=0 2020-12-05T19:06:22.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.296+0000 7f98690e1080 2 removed watcher, disabling cache 2020-12-05T19:06:22.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.300+0000 7f98690e1080 20 remove_watcher() i=1 2020-12-05T19:06:22.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.300+0000 7f98690e1080 20 remove_watcher() i=2 2020-12-05T19:06:22.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.304+0000 7f98690e1080 20 remove_watcher() i=3 2020-12-05T19:06:22.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.308+0000 7f98690e1080 20 remove_watcher() i=4 2020-12-05T19:06:22.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.312+0000 7f98690e1080 20 remove_watcher() i=5 2020-12-05T19:06:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.312+0000 7f98690e1080 20 remove_watcher() i=6 2020-12-05T19:06:22.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.316+0000 7f98690e1080 20 remove_watcher() i=7 2020-12-05T19:06:22.326 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2020-12-05T19:06:22.327 DEBUG:tasks.util.rgw: json result: [] 2020-12-05T19:06:22.327 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-12-05T19:06:22.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-12-05T19:06:22.328 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-12-05T19:06:22.407 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:22.407 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:22.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.416+0000 7f486e689080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-12-05T19:06:22.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.416+0000 7f486e689080 20 check_secure_mon_conn(): mode 1 is insecure 2020-12-05T19:06:22.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.424+0000 7f47a2ffd700 20 reqs_thread_entry: start 2020-12-05T19:06:22.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.424+0000 7f486e689080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.424+0000 7f486e689080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:22.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.424+0000 7f486e689080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.428+0000 7f486e689080 20 rados_obj.operate() r=0 bl.length=855 2020-12-05T19:06:22.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.428+0000 7f486e689080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.428+0000 7f486e689080 20 rados_obj.operate() r=-2 bl.length=0 2020-12-05T19:06:22.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.428+0000 7f486e689080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.428+0000 7f486e689080 20 rados_obj.operate() r=0 bl.length=46 2020-12-05T19:06:22.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.428+0000 7f486e689080 20 rados->read ofs=0 len=0 2020-12-05T19:06:22.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:22.428+0000 7f486e689080 20 rados_obj.operate() r=0 bl.length=358 2020-12-05T19:06:22.442 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"cb3b981d-613a-41b7-a658-0de320ff75a0","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-12-05T19:06:22.442 DEBUG:tasks.util.rgw: json result: {'id': 'cb3b981d-613a-41b7-a658-0de320ff75a0', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-12-05T19:06:22.443 INFO:tasks.rgw:Starting rgw... 2020-12-05T19:06:22.443 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-12-05T19:06:22.444 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-12-05T19:06:22.444 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-12-05T19:06:22.445 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-12-05T19:06:22.447 INFO:tasks.rgw.c1.client.0:Started 2020-12-05T19:06:22.447 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-12-05T19:06:22.448 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-12-05T19:06:22.448 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-12-05T19:06:22.449 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-12-05T19:06:22.451 INFO:tasks.rgw.c1.client.1:Started 2020-12-05T19:06:22.451 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-12-05T19:06:22.452 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-12-05T19:06:22.452 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-12-05T19:06:22.452 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-12-05T19:06:22.485 INFO:tasks.rgw.c2.client.0:Started 2020-12-05T19:06:22.485 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-12-05T19:06:22.486 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-12-05T19:06:22.486 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-12-05T19:06:22.486 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-12-05T19:06:22.489 INFO:tasks.rgw.c2.client.1:Started 2020-12-05T19:06:22.489 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi185.front.sepia.ceph.com:8000/ 2020-12-05T19:06:22.489 INFO:teuthology.orchestra.run.smithi185:> curl http://smithi185.front.sepia.ceph.com:8000/ 2020-12-05T19:06:22.517 INFO:teuthology.orchestra.run.smithi185.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:06:22.517 INFO:teuthology.orchestra.run.smithi185.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:06:22.521 INFO:teuthology.orchestra.run.smithi185.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi185.front.sepia.ceph.com port 8000: Connection refused 2020-12-05T19:06:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-12-05T19:06:23.523 INFO:teuthology.orchestra.run.smithi185:> curl http://smithi185.front.sepia.ceph.com:8000/ 2020-12-05T19:06:23.537 INFO:teuthology.orchestra.run.smithi185.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:06:23.538 INFO:teuthology.orchestra.run.smithi185.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:06:23.539 INFO:teuthology.orchestra.run.smithi185.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2020-12-05T19:06:23.539 INFO:teuthology.orchestra.run.smithi185.stdout:anonymous 2020-12-05T19:06:23.540 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi185.front.sepia.ceph.com:8001/ 2020-12-05T19:06:23.540 INFO:teuthology.orchestra.run.smithi185:> curl http://smithi185.front.sepia.ceph.com:8001/ 2020-12-05T19:06:23.595 INFO:teuthology.orchestra.run.smithi185.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:06:23.595 INFO:teuthology.orchestra.run.smithi185.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:06:23.599 INFO:teuthology.orchestra.run.smithi185.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-12-05T19:06:23.600 INFO:teuthology.orchestra.run.smithi185.stdout:anonymous 2020-12-05T19:06:23.600 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi023.front.sepia.ceph.com:8000/ 2020-12-05T19:06:23.601 INFO:teuthology.orchestra.run.smithi023:> curl http://smithi023.front.sepia.ceph.com:8000/ 2020-12-05T19:06:23.629 INFO:teuthology.orchestra.run.smithi023.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:06:23.630 INFO:teuthology.orchestra.run.smithi023.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:06:23.634 INFO:teuthology.orchestra.run.smithi023.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 17833 0 --:--:-- --:--:-- --:--:-- 42800 2020-12-05T19:06:23.635 INFO:teuthology.orchestra.run.smithi023.stdout:anonymous 2020-12-05T19:06:23.635 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi023.front.sepia.ceph.com:8001/ 2020-12-05T19:06:23.636 INFO:teuthology.orchestra.run.smithi023:> curl http://smithi023.front.sepia.ceph.com:8001/ 2020-12-05T19:06:23.691 INFO:teuthology.orchestra.run.smithi023.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:06:23.691 INFO:teuthology.orchestra.run.smithi023.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:06:23.696 INFO:teuthology.orchestra.run.smithi023.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-12-05T19:06:23.697 INFO:teuthology.orchestra.run.smithi023.stdout:anonymous 2020-12-05T19:06:23.697 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-12-05T19:06:24.033 INFO:tasks.rgw_multisite:creating realm.. 2020-12-05T19:06:24.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:24.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:24.035 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:24.072 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:24.073 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:24.124 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","name":"test-realm","current_period":"fc0acb5f-4a3f-433a-a048-a9d9f51d68ee","epoch":1} 2020-12-05T19:06:24.125 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-12-05T19:06:24.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:24.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:24.126 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:24.203 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:24.204 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:24.228 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:06:24.226+0000 7fcea79b5080 -1 old format 2020-12-05T19:06:24.247 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}} 2020-12-05T19:06:24.247 INFO:tasks.rgw_multisite:creating master zone.. 2020-12-05T19:06:24.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--endpoints', 'http://smithi185.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:24.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--endpoints', 'http://smithi185.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:24.248 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key NFZJ37NSJ0K1XNCM --secret FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO --endpoints http://smithi185.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:24.337 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:24.337 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:24.411 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:06:24.412 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-12-05T19:06:24.833 INFO:teuthology.orchestra.run.smithi185.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-12-05T19:06:24.849 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-12-05T19:06:25.874 INFO:teuthology.orchestra.run.smithi185.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-12-05T19:06:25.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:25.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:25.900 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:25.936 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:25.937 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:25.962 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-12-05T19:06:25.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:25.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:25.964 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:26.045 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:26.045 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:34.114 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"cb156068-23b3-4eca-8803-e19dbeb79445","epoch":1,"predecessor_uuid":"fc0acb5f-4a3f-433a-a048-a9d9f51d68ee","sync_status":[],"period_map":{"id":"cb156068-23b3-4eca-8803-e19dbeb79445","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":2} 2020-12-05T19:06:34.114 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-12-05T19:06:34.115 INFO:teuthology.orchestra.run:waiting for 300 2020-12-05T19:06:34.159 INFO:tasks.rgw.c1.client.0.smithi185.stdout:2020-12-05T19:06:34.158+0000 7fc5166bb700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 11979) UID: 0 2020-12-05T19:06:34.159 INFO:tasks.rgw.c1.client.0.smithi185.stdout:2020-12-05T19:06:34.158+0000 7fc52dd710c0 -1 shutting down 2020-12-05T19:06:37.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-12-05T19:06:40.217 INFO:tasks.rgw.c1.client.0:Stopped 2020-12-05T19:06:40.218 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-12-05T19:06:40.218 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-12-05T19:06:40.221 INFO:tasks.rgw.c1.client.0:Started 2020-12-05T19:06:40.221 INFO:teuthology.orchestra.run.smithi185:> curl http://smithi185.front.sepia.ceph.com:8000 2020-12-05T19:06:40.237 INFO:teuthology.orchestra.run.smithi185.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:06:40.237 INFO:teuthology.orchestra.run.smithi185.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:06:40.241 INFO:teuthology.orchestra.run.smithi185.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi185.front.sepia.ceph.com port 8000: Connection refused 2020-12-05T19:06:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-12-05T19:06:41.243 INFO:teuthology.orchestra.run.smithi185:> curl http://smithi185.front.sepia.ceph.com:8000 2020-12-05T19:06:41.256 INFO:teuthology.orchestra.run.smithi185.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:06:41.256 INFO:teuthology.orchestra.run.smithi185.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:06:41.261 INFO:teuthology.orchestra.run.smithi185.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 53500 2020-12-05T19:06:41.262 INFO:teuthology.orchestra.run.smithi185.stdout:anonymous 2020-12-05T19:06:41.262 INFO:tasks.rgw_multisite:creating admin user.. 2020-12-05T19:06:41.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:41.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:41.263 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key NFZJ37NSJ0K1XNCM --secret FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:41.348 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:41.348 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:41.545 INFO:teuthology.orchestra.run.smithi185.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-12-05T19:06:41.575 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-12-05T19:06:41.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi185.front.sepia.ceph.com:8000', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:41.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi185.front.sepia.ceph.com:8000', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:41.576 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi185.front.sepia.ceph.com:8000 --access-key NFZJ37NSJ0K1XNCM --secret FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:41.617 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:41.618 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:41.748 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","name":"test-realm","current_period":"cb156068-23b3-4eca-8803-e19dbeb79445","epoch":2} 2020-12-05T19:06:41.748 INFO:tasks.rgw_multisite:creating zone.. 2020-12-05T19:06:41.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--endpoints', 'http://smithi023.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:41.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--endpoints', 'http://smithi023.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:41.749 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key NFZJ37NSJ0K1XNCM --secret FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO --endpoints http://smithi023.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:41.836 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:41.836 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:41.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:41.907+0000 7f13e9532080 0 failed reading obj info from .rgw.root:zone_info.e622b2e3-1a91-4dc1-b31e-43ede659e70f: (2) No such file or directory 2020-12-05T19:06:41.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:06:41.907+0000 7f13e9532080 0 WARNING: could not read zone params for zone id=e622b2e3-1a91-4dc1-b31e-43ede659e70f name=test-zone1 2020-12-05T19:06:41.925 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:06:41.926 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-12-05T19:06:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-12-05T19:06:42.705 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-12-05T19:06:43.202 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-12-05T19:06:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-12-05T19:06:43.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:43.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:43.728 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:43.764 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:43.764 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:43.792 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-12-05T19:06:43.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:43.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:06:43.794 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:06:43.869 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:06:43.870 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:06:51.883 INFO:teuthology.orchestra.run.smithi023.stderr:Sending period to new master zone e622b2e3-1a91-4dc1-b31e-43ede659e70f 2020-12-05T19:06:52.983 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"cb156068-23b3-4eca-8803-e19dbeb79445","epoch":2,"predecessor_uuid":"fc0acb5f-4a3f-433a-a048-a9d9f51d68ee","sync_status":[],"period_map":{"id":"cb156068-23b3-4eca-8803-e19dbeb79445","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":2} 2020-12-05T19:06:52.984 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-12-05T19:06:52.985 INFO:teuthology.orchestra.run:waiting for 300 2020-12-05T19:06:53.026 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:06:53.027+0000 7f08ae22d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 11809) UID: 0 2020-12-05T19:06:53.026 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:06:53.027+0000 7f08c58e30c0 -1 shutting down 2020-12-05T19:06:58.276 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-12-05T19:06:59.087 INFO:tasks.rgw.c2.client.0:Stopped 2020-12-05T19:06:59.087 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-12-05T19:06:59.088 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-12-05T19:06:59.090 INFO:tasks.rgw.c2.client.0:Started 2020-12-05T19:06:59.091 INFO:teuthology.orchestra.run.smithi023:> curl http://smithi023.front.sepia.ceph.com:8000 2020-12-05T19:06:59.107 INFO:teuthology.orchestra.run.smithi023.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:06:59.108 INFO:teuthology.orchestra.run.smithi023.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:06:59.112 INFO:teuthology.orchestra.run.smithi023.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi023.front.sepia.ceph.com port 8000: Connection refused 2020-12-05T19:06:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-12-05T19:07:00.114 INFO:teuthology.orchestra.run.smithi023:> curl http://smithi023.front.sepia.ceph.com:8000 2020-12-05T19:07:00.130 INFO:teuthology.orchestra.run.smithi023.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:07:00.131 INFO:teuthology.orchestra.run.smithi023.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:07:00.132 INFO:teuthology.orchestra.run.smithi023.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-12-05T19:07:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:anonymous 2020-12-05T19:07:00.134 INFO:tasks.rgw_multisite:creating zone.. 2020-12-05T19:07:00.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--endpoints', 'http://smithi185.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:00.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--endpoints', 'http://smithi185.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:00.135 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key NFZJ37NSJ0K1XNCM --secret FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO --endpoints http://smithi185.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:07:00.180 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:00.180 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:00.238 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:07:00.238+0000 7fb0fd96f080 0 failed reading obj info from .rgw.root:zone_info.3ee88d97-6118-4d9f-87f0-34d91092c64f: (2) No such file or directory 2020-12-05T19:07:00.238 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:07:00.238+0000 7fb0fd96f080 0 WARNING: could not read zone params for zone id=3ee88d97-6118-4d9f-87f0-34d91092c64f name=test-zone2 2020-12-05T19:07:00.250 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:07:00.251 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-12-05T19:07:00.867 INFO:teuthology.orchestra.run.smithi185.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-12-05T19:07:00.886 INFO:teuthology.orchestra.run.smithi185:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-12-05T19:07:01.868 INFO:teuthology.orchestra.run.smithi185.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-12-05T19:07:01.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:01.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:01.892 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:07:01.952 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:01.952 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:01.977 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-12-05T19:07:01.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:01.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:01.978 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:07:02.012 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:02.012 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:03.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-12-05T19:07:10.082 INFO:teuthology.orchestra.run.smithi185.stderr:Sending period to new master zone e622b2e3-1a91-4dc1-b31e-43ede659e70f 2020-12-05T19:07:11.184 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"cb156068-23b3-4eca-8803-e19dbeb79445","epoch":3,"predecessor_uuid":"fc0acb5f-4a3f-433a-a048-a9d9f51d68ee","sync_status":[],"period_map":{"id":"cb156068-23b3-4eca-8803-e19dbeb79445","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":2} 2020-12-05T19:07:11.185 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-12-05T19:07:11.185 INFO:teuthology.orchestra.run:waiting for 300 2020-12-05T19:07:11.229 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:07:11.230+0000 7fab3a7fc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 11983) UID: 0 2020-12-05T19:07:11.229 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:07:11.230+0000 7fab5df510c0 -1 shutting down 2020-12-05T19:07:17.287 INFO:tasks.rgw.c1.client.1:Stopped 2020-12-05T19:07:17.288 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-12-05T19:07:17.288 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-12-05T19:07:17.291 INFO:tasks.rgw.c1.client.1:Started 2020-12-05T19:07:17.291 INFO:teuthology.orchestra.run.smithi185:> curl http://smithi185.front.sepia.ceph.com:8001 2020-12-05T19:07:17.307 INFO:teuthology.orchestra.run.smithi185.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:07:17.308 INFO:teuthology.orchestra.run.smithi185.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:07:17.311 INFO:teuthology.orchestra.run.smithi185.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi185.front.sepia.ceph.com port 8001: Connection refused 2020-12-05T19:07:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-12-05T19:07:17.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-12-05T19:07:18.313 INFO:teuthology.orchestra.run.smithi185:> curl http://smithi185.front.sepia.ceph.com:8001 2020-12-05T19:07:18.328 INFO:teuthology.orchestra.run.smithi185.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:07:18.329 INFO:teuthology.orchestra.run.smithi185.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:07:18.333 INFO:teuthology.orchestra.run.smithi185.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-12-05T19:07:18.334 INFO:teuthology.orchestra.run.smithi185.stdout:anonymous 2020-12-05T19:07:18.334 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-12-05T19:07:18.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi185.front.sepia.ceph.com:8000', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--url', 'http://smithi185.front.sepia.ceph.com:8000', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:18.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi185.front.sepia.ceph.com:8000', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--url', 'http://smithi185.front.sepia.ceph.com:8000', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:18.335 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi185.front.sepia.ceph.com:8000 --access-key NFZJ37NSJ0K1XNCM --secret FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO --url http://smithi185.front.sepia.ceph.com:8000 --access-key NFZJ37NSJ0K1XNCM --secret FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:07:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:18.435 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:07:18.435+0000 7f433f606700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.4217a5cb-deb1-4741-a893-a42d4fc447ab.control err=-107 2020-12-05T19:07:18.460 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","name":"test-realm","current_period":"cb156068-23b3-4eca-8803-e19dbeb79445","epoch":2} 2020-12-05T19:07:18.460 INFO:tasks.rgw_multisite:creating zone.. 2020-12-05T19:07:18.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--endpoints', 'http://smithi023.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:18.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--endpoints', 'http://smithi023.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:18.462 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key NFZJ37NSJ0K1XNCM --secret FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO --endpoints http://smithi023.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:07:18.513 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:18.513 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:18.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:07:18.575+0000 7f604bdcb080 0 failed reading obj info from .rgw.root:zone_info.e622b2e3-1a91-4dc1-b31e-43ede659e70f: (2) No such file or directory 2020-12-05T19:07:18.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:07:18.575+0000 7f604bdcb080 0 WARNING: could not read zone params for zone id=e622b2e3-1a91-4dc1-b31e-43ede659e70f name=test-zone1 2020-12-05T19:07:18.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:07:18.575+0000 7f604bdcb080 0 failed reading obj info from .rgw.root:zone_info.ecced906-ad37-41c3-ad42-864c1e77463f: (2) No such file or directory 2020-12-05T19:07:18.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:07:18.575+0000 7f604bdcb080 0 WARNING: could not read zone params for zone id=ecced906-ad37-41c3-ad42-864c1e77463f name=test-zone3 2020-12-05T19:07:18.587 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:07:18.588 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-12-05T19:07:19.692 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-12-05T19:07:19.708 INFO:teuthology.orchestra.run.smithi023:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-12-05T19:07:20.689 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-12-05T19:07:20.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:20.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:20.708 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:07:20.760 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:20.760 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:20.783 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":""} 2020-12-05T19:07:20.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:20.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:20.784 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:07:20.818 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:20.818 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:23.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-12-05T19:07:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:Sending period to new master zone e622b2e3-1a91-4dc1-b31e-43ede659e70f 2020-12-05T19:07:30.154 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"cb156068-23b3-4eca-8803-e19dbeb79445","epoch":4,"predecessor_uuid":"fc0acb5f-4a3f-433a-a048-a9d9f51d68ee","sync_status":[],"period_map":{"id":"cb156068-23b3-4eca-8803-e19dbeb79445","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":2} 2020-12-05T19:07:30.155 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-12-05T19:07:30.156 INFO:teuthology.orchestra.run:waiting for 300 2020-12-05T19:07:30.197 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:07:30.199+0000 7fc3027fc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 11812) UID: 0 2020-12-05T19:07:30.197 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:07:30.199+0000 7fc3262cf0c0 -1 shutting down 2020-12-05T19:07:32.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-12-05T19:07:36.258 INFO:tasks.rgw.c2.client.1:Stopped 2020-12-05T19:07:36.258 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-12-05T19:07:36.259 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-12-05T19:07:36.261 INFO:tasks.rgw.c2.client.1:Started 2020-12-05T19:07:36.262 INFO:teuthology.orchestra.run.smithi023:> curl http://smithi023.front.sepia.ceph.com:8001 2020-12-05T19:07:36.279 INFO:teuthology.orchestra.run.smithi023.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:07:36.279 INFO:teuthology.orchestra.run.smithi023.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:07:36.283 INFO:teuthology.orchestra.run.smithi023.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi023.front.sepia.ceph.com port 8001: Connection refused 2020-12-05T19:07:36.284 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-12-05T19:07:37.285 INFO:teuthology.orchestra.run.smithi023:> curl http://smithi023.front.sepia.ceph.com:8001 2020-12-05T19:07:37.300 INFO:teuthology.orchestra.run.smithi023.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:07:37.301 INFO:teuthology.orchestra.run.smithi023.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:07:37.305 INFO:teuthology.orchestra.run.smithi023.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-12-05T19:07:37.306 INFO:teuthology.orchestra.run.smithi023.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005fcbda79-10e7-test-zone4","HostId":"10e7-test-zone4-test-zonegroup"} 2020-12-05T19:07:37.306 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-12-05T19:07:37.307 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-12-05T19:07:37.361 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-12-05T19:07:37.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:37.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:07:37.363 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:07:37.406 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:37.406 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:37.620 INFO:teuthology.orchestra.run.smithi185.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"GFCZJ6KBD0OZIRLWKKKW","secret_key":"mhDfRRqOaFYD2M6EDwIzAz9I1trJ5Sg9Xbdir6H0"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-12-05T19:07:37.650 INFO:rgw_multi.tests:meta checkpoint 2020-12-05T19:07:37.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:37.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:37.652 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:07:37.731 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:37.731 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:37.911 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:07:37.939 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195257.616573_45.1","last_update":"2020-12-05T19:07:37.616573Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:07:37.939 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:07:37.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:37.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:37.941 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:07:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:38.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-12-05T19:07:38.206 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:07:38.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:07:43.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:43.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:43.209 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:07:43.251 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:43.251 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:43.480 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:07:43.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:07:48.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:48.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:48.483 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:07:48.524 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:48.524 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:48.758 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:07:48.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:07:53.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:53.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:53.761 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:07:53.799 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:53.799 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:54.032 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:07:54.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:07:59.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:59.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:07:59.034 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:07:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:07:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:07:59.321 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:07:59.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:04.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:04.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:04.324 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:04.365 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:04.365 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:04.604 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:04.605 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:09.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:09.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:09.607 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:09.648 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:09.648 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:09.880 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:09.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:14.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:14.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:14.883 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:14.924 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:14.924 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:15.164 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:15.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:20.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:20.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:20.167 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:20.209 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:20.209 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:20.459 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:20.460 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:25.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:25.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:25.462 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:25.503 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:25.504 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:25.747 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:25.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:30.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:30.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:30.751 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:31.023 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:31.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:36.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:36.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:36.026 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:36.067 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:36.068 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:36.309 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:36.310 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:41.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:41.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:41.312 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:41.355 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:41.355 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:41.578 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:41.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:46.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:46.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:46.581 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:46.836 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:46.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:51.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:51.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:51.842 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:51.883 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:51.883 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:52.137 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:52.138 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:08:57.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:57.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:08:57.140 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:08:57.180 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:08:57.180 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:08:57.412 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:08:57.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:09:02.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:02.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:02.414 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:02.453 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:02.453 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:02.707 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195257.616573_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:07:37.616573Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:02.729 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:09:02.729 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:09:02.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:02.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:02.730 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:02.772 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:02.772 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:02.995 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:02.996 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:09:07.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:07.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:07.998 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:08.038 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:08.039 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:08.265 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:08.265 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:09:13.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:13.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:13.267 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:13.307 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:13.307 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:13.522 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:13.523 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:09:18.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:18.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:18.525 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:18.565 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:18.565 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:18.782 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:18.783 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:09:23.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:23.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:23.785 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:23.823 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:23.823 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:24.047 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:24.048 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:09:29.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:29.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:29.050 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:29.091 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:29.091 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:29.322 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:29.323 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1607195257.616573_45.1 target= 2020-12-05T19:09:34.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:34.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:34.325 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:34.366 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:34.366 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:34.568 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195257.616573_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:07:37.616573Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:34.568 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:09:34.569 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:09:34.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:34.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:34.570 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:34.610 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:34.610 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:34.878 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195257.616573_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:07:37.616573Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:34.879 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:09:34.879 INFO:rgw_multi.tests:meta checkpoint 2020-12-05T19:09:34.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:34.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:34.880 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:34.915 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:34.915 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:35.091 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:09:35.128 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195257.616573_45.1","last_update":"2020-12-05T19:07:37.616573Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:09:35.128 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:09:35.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:35.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:35.129 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:35.170 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:35.170 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:35.427 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195257.616573_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:07:37.616573Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:35.428 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:09:35.428 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:09:35.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:35.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:35.429 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:35.468 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:35.469 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:35.689 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195257.616573_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:07:37.616573Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:35.689 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:09:35.690 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:09:35.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:35.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:35.692 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:35.732 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:35.733 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:36.009 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195257.616573_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:07:37.616573Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:36.010 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:09:36.010 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-12-05T19:09:36.016 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-1 2020-12-05T19:09:40.021 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-2 2020-12-05T19:09:43.658 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-3 2020-12-05T19:09:43.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:43.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:43.758 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:43.793 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:43.794 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:43.969 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:09:43.998 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195383.731707_57.1","last_update":"2020-12-05T19:09:43.731707Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:09:43.999 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:09:44.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:44.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:44.001 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:44.042 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:44.043 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:44.292 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195380.057318_53.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:09:40.057318Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:44.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195383.731707_57.1 target=1_1607195380.057318_53.1 2020-12-05T19:09:49.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:49.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:49.295 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:49.338 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:49.338 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:49.564 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195383.731707_57.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:09:43.731707Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:49.565 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:09:49.565 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:09:49.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:49.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:49.567 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:49.607 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:49.607 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:49.835 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195383.731707_57.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:09:43.731707Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:49.835 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:09:49.836 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:09:49.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:49.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:49.837 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:49.878 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:49.879 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:50.151 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195383.731707_57.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:09:43.731707Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:50.151 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:09:51.316 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-12-05T19:09:51.317 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-4 2020-12-05T19:09:51.349 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-5 2020-12-05T19:09:51.409 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-6 2020-12-05T19:09:51.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:51.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:51.467 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:51.502 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:51.502 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:51.683 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:09:51.716 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195391.448654_71.1","last_update":"2020-12-05T19:09:51.448654Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:09:51.717 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:09:51.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:51.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:51.719 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:51.760 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:51.760 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:51.999 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195383.731707_57.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:09:43.731707Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:52.000 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195391.448654_71.1 target=1_1607195383.731707_57.1 2020-12-05T19:09:57.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:57.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:09:57.003 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:09:57.045 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:09:57.045 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:09:57.286 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195391.441771_69.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:09:51.441771Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:09:57.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195391.448654_71.1 target=1_1607195391.441771_69.1 2020-12-05T19:10:02.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:02.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:02.289 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:02.330 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:02.331 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:02.543 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195391.448654_71.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:09:51.448654Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:02.544 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:10:02.544 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:10:02.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:02.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:02.546 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:02.587 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:02.587 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:02.804 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195391.448654_71.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:09:51.448654Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:02.804 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:10:02.805 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:10:02.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:02.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:02.806 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:02.847 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:02.847 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:03.116 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195391.448654_71.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:09:51.448654Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:03.116 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:10:04.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:04.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:04.822 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:04.859 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:04.859 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:05.047 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:10:05.082 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195404.697154_107.1","last_update":"2020-12-05T19:10:04.697154Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:10:05.082 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:10:05.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:05.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:05.084 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:05.126 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:05.127 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:05.355 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195404.286241_91.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:04.286241Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:05.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195404.697154_107.1 target=1_1607195404.286241_91.1 2020-12-05T19:10:10.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:10.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:10.357 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:10.397 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:10.398 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:10.645 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195404.577361_101.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:04.577361Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:10.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195404.697154_107.1 target=1_1607195404.577361_101.1 2020-12-05T19:10:15.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:15.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:15.649 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:15.692 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:15.692 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:15.920 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195404.697154_107.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:04.697154Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:15.920 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:10:15.921 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:10:15.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:15.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:15.922 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:15.955 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:15.955 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:16.198 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195404.697154_107.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:04.697154Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:16.199 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:10:16.200 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:10:16.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:16.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:16.201 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:16.240 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:16.514 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195404.697154_107.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:04.697154Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:16.514 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:10:17.948 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-12-05T19:10:17.950 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-7 2020-12-05T19:10:17.974 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-8 2020-12-05T19:10:18.031 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-9 2020-12-05T19:10:18.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:18.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:18.085 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:18.120 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:18.121 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:18.327 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:10:18.364 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195418.067947_119.1","last_update":"2020-12-05T19:10:18.067947Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:10:18.364 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:10:18.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:18.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:18.366 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:18.405 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:18.405 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:18.649 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195404.697154_107.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:04.697154Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:18.649 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195418.067947_119.1 target=1_1607195404.697154_107.1 2020-12-05T19:10:23.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:23.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:23.652 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:23.692 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:23.693 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:23.920 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195418.067947_119.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:18.067947Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:23.921 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:10:23.922 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:10:23.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:23.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:23.923 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:23.963 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:23.963 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:24.199 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195418.067947_119.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:18.067947Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:24.199 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:10:24.200 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:10:24.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:24.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:24.203 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:24.244 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:24.244 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:24.512 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195418.067947_119.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:18.067947Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:24.513 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:10:26.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:26.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:26.404 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:26.439 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:26.440 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:26.638 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:10:26.673 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:10:26.674 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:10:26.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:26.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:26.675 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:26.717 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:26.717 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:26.976 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.284587_123.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.284587Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:26.976 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1607195426.378999_125.1 target=1_1607195426.284587_123.1 2020-12-05T19:10:31.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:31.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:31.979 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:32.019 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:32.020 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:32.282 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:32.283 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:10:32.283 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:10:32.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:32.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:32.285 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:32.325 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:32.325 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:32.551 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:32.552 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:10:32.553 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:10:32.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:32.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:32.556 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:32.596 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:32.597 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:32.872 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:32.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:10:33.434 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-12-05T19:10:33.435 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-10 2020-12-05T19:10:33.473 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-11 2020-12-05T19:10:33.536 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-12 2020-12-05T19:10:34.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:34.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:34.208 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:34.247 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:34.247 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:34.463 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:10:34.502 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195433.575954_137.1","last_update":"2020-12-05T19:10:33.575954Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:10:34.503 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:10:34.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:34.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:34.504 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:34.548 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:34.549 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:34.757 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195433.568424_135.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:10:33.568424Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:34.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195433.575954_137.1 target=1_1607195433.568424_135.1 2020-12-05T19:10:39.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:39.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:39.760 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:39.800 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:39.801 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:40.047 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195433.575954_137.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:10:33.575954Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:40.048 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:10:40.048 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:10:40.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:40.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:40.049 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:40.090 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:40.091 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:40.332 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195433.575954_137.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:10:33.575954Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:40.333 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:10:40.333 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:10:40.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:40.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:40.335 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:40.660 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195433.575954_137.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:10:33.575954Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:40.660 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:10:40.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:40.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:40.662 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:40.705 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:40.706 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:40.931 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.14.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.9.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.9.5"},{"key":8,"val":"00000000001.19.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:10:40.931 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-10 2020-12-05T19:10:40.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:40.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:40.933 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:40.977 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:40.977 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:41.231 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.19.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:10:41.232 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-10 2020-12-05T19:10:41.232 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-10 zones={test-zone2, test-zone1} 2020-12-05T19:10:42.036 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:10:42.044 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:42.045 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:42.045 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:42.046 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:42.046 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:10:42.053 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:42.053 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:42.054 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:42.054 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:42.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:42.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:42.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:42.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:42.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:42.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:42.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:42.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:42.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:42.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:42.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:43.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:43.150 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-10 zones={test-zone2, test-zone1} 2020-12-05T19:10:43.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:43.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:43.152 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:43.190 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:43.190 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:43.415 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.14.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.9.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.9.5"},{"key":8,"val":"00000000001.19.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:10:43.416 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-10 2020-12-05T19:10:43.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:43.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:43.418 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:43.501 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:43.501 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:43.743 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.19.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:10:43.744 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-10 2020-12-05T19:10:43.744 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-10 zones={test-zone3, test-zone1} 2020-12-05T19:10:43.954 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:10:43.960 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:43.960 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:43.961 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:43.961 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:43.961 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:10:43.966 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:43.967 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:43.967 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:43.967 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:43.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:44.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:44.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:44.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:44.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:44.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:44.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:44.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:44.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:44.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:44.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:44.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:45.103 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-10 zones={test-zone3, test-zone1} 2020-12-05T19:10:45.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:45.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:45.104 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:45.140 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:45.141 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:45.355 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.14.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.9.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.9.5"},{"key":8,"val":"00000000001.19.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:10:45.356 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-10 2020-12-05T19:10:45.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:45.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:45.357 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:45.402 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:45.403 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:45.676 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.14.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.9.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.9.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.19.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:10:45.676 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-10 2020-12-05T19:10:45.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:45.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:45.678 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:45.754 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:45.754 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:45.986 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.4.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.16.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.10.5"},{"key":8,"val":"00000000001.6.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:10:45.987 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-11 2020-12-05T19:10:45.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:45.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:45.988 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:46.033 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:46.033 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:46.288 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.4.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:10:46.289 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-11 2020-12-05T19:10:46.290 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-11 zones={test-zone1, test-zone2} 2020-12-05T19:10:47.225 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:10:47.235 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:47.236 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:47.236 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:47.237 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:47.237 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:10:47.244 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:47.245 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:47.246 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:47.246 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:47.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:47.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:47.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:47.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:47.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:47.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:47.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:47.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:47.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:48.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:48.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:48.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:48.361 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-11 zones={test-zone1, test-zone2} 2020-12-05T19:10:48.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:48.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:48.363 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:48.401 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:48.641 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.4.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.16.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.10.5"},{"key":8,"val":"00000000001.6.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:10:48.642 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-11 2020-12-05T19:10:48.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:48.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:48.645 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:48.691 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:48.691 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:48.943 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.4.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:10:48.944 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-11 2020-12-05T19:10:48.944 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-11 zones={test-zone3, test-zone2} 2020-12-05T19:10:49.317 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:10:49.325 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:49.325 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:49.326 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:49.326 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:49.327 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:10:49.334 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:49.334 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:49.335 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:49.335 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:49.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:49.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:49.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:49.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:49.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:49.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:49.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:49.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:50.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:50.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:50.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:50.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:50.432 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-11 zones={test-zone3, test-zone2} 2020-12-05T19:10:50.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:50.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:50.434 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:50.476 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:50.476 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:50.708 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.6.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.6.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.3.5"},{"key":8,"val":"00000000001.8.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:10:50.708 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-12 2020-12-05T19:10:50.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:50.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:50.710 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:50.797 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:50.797 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:51.040 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.3.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:10:51.041 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-12 2020-12-05T19:10:51.041 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-12 zones={test-zone1, test-zone3} 2020-12-05T19:10:51.346 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:10:51.351 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:51.352 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:51.352 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:51.352 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:51.353 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:10:51.358 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:51.359 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:51.359 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:51.360 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:51.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:51.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:51.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:51.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:51.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:51.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:51.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:51.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:52.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:52.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:52.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:52.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:52.500 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-12 zones={test-zone1, test-zone3} 2020-12-05T19:10:52.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:52.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:52.501 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:52.539 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:52.539 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:52.774 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.6.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.6.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.3.5"},{"key":8,"val":"00000000001.8.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:10:52.775 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-12 2020-12-05T19:10:52.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:52.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:52.777 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:53.070 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.3.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:10:53.071 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-12 2020-12-05T19:10:53.071 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-12 zones={test-zone2, test-zone3} 2020-12-05T19:10:53.191 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:10:53.197 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:53.198 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:53.198 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:53.199 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:53.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:10:53.206 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-12-05T19:10:53.207 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-12-05T19:10:53.207 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-12-05T19:10:53.208 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:10:53.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:53.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:53.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-12-05T19:10:53.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:53.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:53.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-12-05T19:10:53.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:53.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:53.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-12-05T19:10:54.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:54.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:54.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:10:54.329 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-12 zones={test-zone2, test-zone3} 2020-12-05T19:10:54.329 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-12-05T19:10:54.330 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-13 2020-12-05T19:10:54.953 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-14 2020-12-05T19:10:55.005 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-15 2020-12-05T19:10:55.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:55.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:55.208 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:55.247 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:55.248 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:55.434 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:10:55.468 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195455.044591_159.1","last_update":"2020-12-05T19:10:55.044591Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:10:55.469 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:10:55.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:55.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:10:55.471 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:10:55.516 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:10:55.517 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:10:55.728 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195455.038390_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:10:55.038390Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:10:55.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195455.044591_159.1 target=1_1607195455.038390_157.1 2020-12-05T19:11:00.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:00.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:00.732 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:00.774 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:00.774 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:00.997 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195455.044591_159.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:10:55.044591Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:11:00.998 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:11:00.999 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:11:00.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:01.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:01.000 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:01.041 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:01.042 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:01.257 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195455.044591_159.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:10:55.044591Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:11:01.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:11:01.259 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:11:01.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:01.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:01.262 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:01.303 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:01.304 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:01.575 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195455.044591_159.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:10:55.044591Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:11:01.575 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:11:01.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:01.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:01.577 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:01.618 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:01.618 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:01.845 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:01.846 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:01.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:01.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:01.848 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:01.892 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:01.893 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:02.136 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:02.136 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:02.137 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-13 zones={test-zone2, test-zone1} 2020-12-05T19:11:03.214 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:03.221 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:03.222 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:03.270 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:03.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:03.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:03.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:03.611 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-13 zones={test-zone2, test-zone1} 2020-12-05T19:11:03.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:03.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:03.613 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:03.650 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:03.651 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:03.878 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:03.879 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:03.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:03.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:03.880 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:03.957 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:03.957 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:04.189 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:04.190 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:04.190 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-13 zones={test-zone3, test-zone1} 2020-12-05T19:11:05.252 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:05.260 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:05.261 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:05.310 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:05.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:05.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:05.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:05.662 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-13 zones={test-zone3, test-zone1} 2020-12-05T19:11:05.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:05.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:05.663 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:05.700 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:05.701 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:05.916 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:05.917 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:05.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:05.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:05.918 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:06.247 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:06.248 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:06.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:06.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:06.249 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:06.290 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:06.291 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:06.510 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.20.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:06.510 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-14 2020-12-05T19:11:06.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:06.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:06.512 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:06.556 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:06.557 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:06.806 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.20.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:06.806 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-14 2020-12-05T19:11:06.807 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-14 zones={test-zone1, test-zone2} 2020-12-05T19:11:08.265 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:08.274 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:08.274 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:08.322 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:08.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:08.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:08.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:08.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-14 zones={test-zone1, test-zone2} 2020-12-05T19:11:08.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:08.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:08.656 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:08.697 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:08.697 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:08.915 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.20.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:08.915 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-14 2020-12-05T19:11:08.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:08.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:08.917 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:08.961 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:08.961 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:09.198 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.20.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:09.199 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-14 2020-12-05T19:11:09.199 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-14 zones={test-zone3, test-zone2} 2020-12-05T19:11:10.472 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:10.481 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:10.482 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:10.530 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:10.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:10.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:10.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:10.873 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-14 zones={test-zone3, test-zone2} 2020-12-05T19:11:10.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:10.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:10.875 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:10.911 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:10.912 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:11.128 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.6.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:11.128 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-15 2020-12-05T19:11:11.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:11.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:11.130 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:11.216 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:11.216 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:11.458 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:11.459 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-15 2020-12-05T19:11:11.459 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-15 zones={test-zone1, test-zone3} 2020-12-05T19:11:13.078 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:13.087 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:13.088 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:13.134 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:13.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:13.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:13.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:13.467 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-15 zones={test-zone1, test-zone3} 2020-12-05T19:11:13.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:13.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:13.468 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:13.505 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:13.506 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:13.715 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.6.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:13.715 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-15 2020-12-05T19:11:13.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:13.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:13.717 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:13.761 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:13.761 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:14.016 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:14.016 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-15 2020-12-05T19:11:14.017 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-15 zones={test-zone2, test-zone3} 2020-12-05T19:11:15.688 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:15.696 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:15.696 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:15.742 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-12-05T19:11:15.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:15.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:15.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-12-05T19:11:16.083 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-15 zones={test-zone2, test-zone3} 2020-12-05T19:11:16.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:16.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:16.766 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:16.804 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:16.805 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:17.015 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:17.016 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:17.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:17.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:17.017 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:17.061 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:17.062 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:17.308 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:17.309 WARNING:rgw_multi.tests:bucket tazcyr-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.9.5 target=00000000001.7.5 2020-12-05T19:11:22.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:22.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:22.311 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:22.355 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:22.356 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:22.617 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:22.617 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-13 zones={test-zone2, test-zone1} 2020-12-05T19:11:23.564 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:23.571 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:23.686 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-13 zones={test-zone2, test-zone1} 2020-12-05T19:11:23.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:23.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:23.688 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:23.729 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:23.729 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:23.952 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:23.953 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:23.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:23.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:23.955 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:24.035 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:24.036 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:24.275 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:24.276 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:24.276 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-13 zones={test-zone3, test-zone1} 2020-12-05T19:11:25.107 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:25.115 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:25.218 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-13 zones={test-zone3, test-zone1} 2020-12-05T19:11:25.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:25.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:25.220 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:25.261 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:25.262 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:25.481 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:25.481 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:25.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:25.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:25.483 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:25.527 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:25.528 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:25.809 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:25.810 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-13 2020-12-05T19:11:25.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:25.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:25.851 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:25.890 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:25.890 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:26.131 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.24.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:26.132 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-14 2020-12-05T19:11:26.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:26.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:26.134 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:26.180 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:26.180 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:26.416 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.20.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:26.417 WARNING:rgw_multi.tests:bucket tazcyr-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.24.5 target=00000000001.20.5 2020-12-05T19:11:31.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:31.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:31.420 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:31.465 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:31.466 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:31.710 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.24.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:31.711 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-14 2020-12-05T19:11:31.711 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-14 zones={test-zone1, test-zone2} 2020-12-05T19:11:32.758 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:32.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:32.870 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-14 zones={test-zone1, test-zone2} 2020-12-05T19:11:32.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:32.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:32.873 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:33.137 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.24.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:33.137 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-14 2020-12-05T19:11:33.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:33.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:33.139 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:33.183 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:33.183 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:33.420 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.24.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:33.420 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-14 2020-12-05T19:11:33.421 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-14 zones={test-zone3, test-zone2} 2020-12-05T19:11:35.079 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:35.088 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:35.190 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-14 zones={test-zone3, test-zone2} 2020-12-05T19:11:35.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:35.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:35.213 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:35.250 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:35.251 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:35.459 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:35.460 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-15 2020-12-05T19:11:35.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:35.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:35.462 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:35.546 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:35.546 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:35.791 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:35.792 WARNING:rgw_multi.tests:bucket tazcyr-15 zone test-zone1 behind zone test-zone3: shard=1 master=00000000002.9.5 target=00000000001.6.5 2020-12-05T19:11:40.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:40.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:40.795 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:40.840 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:40.841 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:41.067 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:41.068 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-15 2020-12-05T19:11:41.068 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-15 zones={test-zone1, test-zone3} 2020-12-05T19:11:42.914 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:42.921 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:43.026 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-15 zones={test-zone1, test-zone3} 2020-12-05T19:11:43.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:43.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:43.028 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:43.069 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:43.069 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:43.294 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:43.295 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-15 2020-12-05T19:11:43.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:43.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:43.296 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:43.342 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:43.342 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:43.590 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:43.591 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-15 2020-12-05T19:11:43.591 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-15 zones={test-zone2, test-zone3} 2020-12-05T19:11:43.890 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:11:43.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:11:44.005 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-15 zones={test-zone2, test-zone3} 2020-12-05T19:11:44.006 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-12-05T19:11:44.007 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-16 2020-12-05T19:11:44.161 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-17 2020-12-05T19:11:44.216 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-18 2020-12-05T19:11:44.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:44.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:44.313 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:44.349 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:44.350 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:44.534 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:11:44.568 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195504.311460_197.1","last_update":"2020-12-05T19:11:44.311460Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:11:44.568 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:11:44.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:44.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:44.570 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:44.614 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:44.614 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:44.845 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195455.044591_159.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:10:55.044591Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:11:44.845 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195504.311460_197.1 target=1_1607195455.044591_159.1 2020-12-05T19:11:49.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:49.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:49.848 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:49.887 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:49.887 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:50.109 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195504.249765_189.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:11:44.249765Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:11:50.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195504.311460_197.1 target=1_1607195504.249765_189.1 2020-12-05T19:11:55.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:55.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:55.112 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:55.154 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:55.154 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:55.380 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195504.311460_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:11:44.311460Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:11:55.380 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:11:55.381 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:11:55.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:55.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:55.383 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:55.422 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:55.423 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:55.646 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195504.311460_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:11:44.311460Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:11:55.646 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:11:55.647 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:11:55.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:55.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:55.648 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:55.956 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195504.311460_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:11:44.311460Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:11:55.957 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:11:56.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:56.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:56.496 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:56.536 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:56.536 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:56.757 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.26.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:11:56.758 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:11:56.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:56.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:11:56.760 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:11:56.803 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:11:56.804 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:11:57.043 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:11:57.044 WARNING:rgw_multi.tests:bucket tazcyr-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.26.10 target= 2020-12-05T19:12:02.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:02.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:02.046 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:02.090 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:02.091 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:02.330 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000005.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:02.331 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:02.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:02.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:02.333 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:02.373 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:02.374 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:02.592 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.32.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:02.592 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:02.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:02.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:02.594 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:02.676 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:02.677 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:02.913 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000005.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:02.913 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:02.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:02.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:02.915 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:02.996 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:02.997 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:03.225 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.32.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:03.226 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:03.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:03.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:03.228 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:03.272 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:03.272 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:03.544 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:03.545 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:03.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:03.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:03.784 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:03.823 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:03.824 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:04.052 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:04.053 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:12:04.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:04.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:04.054 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:04.097 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:04.097 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:04.329 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:04.330 WARNING:rgw_multi.tests:bucket tazcyr-17 zone test-zone1 behind zone test-zone2: shard=5 master=00000000002.14.10 target= 2020-12-05T19:12:09.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:09.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:09.332 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:09.377 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:09.377 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:09.632 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000006.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:09.632 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:12:09.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:09.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:09.634 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:09.677 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:09.884 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:12:09.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:09.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:09.886 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:09.933 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:09.934 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:10.164 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000006.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:10.164 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:12:10.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:10.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:10.700 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:10.740 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:10.741 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:10.962 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:10.963 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:12:10.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:10.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:10.965 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:11.045 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:11.046 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:11.308 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:11.309 WARNING:rgw_multi.tests:bucket tazcyr-18 zone test-zone1 behind zone test-zone3: shard=5 master=00000000002.14.10 target= 2020-12-05T19:12:16.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:16.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:16.312 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:16.353 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:16.354 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:16.605 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:16.606 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:12:16.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:16.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:16.609 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:16.688 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:16.688 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:16.920 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:16.920 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:12:16.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:16.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:16.922 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:16.966 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:16.967 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:17.216 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:17.217 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:12:20.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:20.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:20.091 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:20.132 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:20.132 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:20.355 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000002.14.10"},{"key":4,"val":""},{"key":5,"val":"00000000007.36.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:20.355 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:20.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:20.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:20.357 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:20.404 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:20.404 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:20.627 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000005.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:20.628 WARNING:rgw_multi.tests:bucket tazcyr-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000002.14.10 target=, shard=5 master=00000000007.36.10 target=00000000005.32.9 2020-12-05T19:12:25.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:25.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:25.631 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:25.912 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000008.38.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:25.912 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:25.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:25.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:25.914 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:25.956 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:25.956 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:26.193 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.38.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:26.194 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:26.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:26.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:26.195 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:26.276 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:26.277 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:26.527 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000008.38.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:26.528 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:26.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:26.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:26.530 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:26.611 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:26.611 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:26.851 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.38.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:26.852 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:26.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:26.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:26.853 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:26.897 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:26.898 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:27.173 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.38.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:27.174 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:27.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:27.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:27.176 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:27.255 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:27.255 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:27.480 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.19.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.26.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:27.480 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-16 2020-12-05T19:12:27.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:27.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:27.482 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:27.527 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:27.527 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:27.790 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000010.26.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000007.26.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:27.791 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-16 2020-12-05T19:12:27.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:27.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:27.792 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:27.832 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:27.833 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:28.082 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.19.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.26.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:28.083 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-16 2020-12-05T19:12:28.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:28.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:28.085 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:28.130 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:28.130 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:28.381 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000010.26.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000007.26.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:28.382 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-16 2020-12-05T19:12:28.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:28.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:28.383 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:28.463 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:28.463 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:28.696 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.10.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.42.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:28.696 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-16 2020-12-05T19:12:28.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:28.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:28.698 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:28.780 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:28.781 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:29.049 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000003.10.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000014.42.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:29.049 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-16 2020-12-05T19:12:29.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:29.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:29.051 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:29.133 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:29.134 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:29.361 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.10.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.42.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:29.361 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-16 2020-12-05T19:12:29.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:29.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:29.363 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:29.407 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:29.407 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:29.673 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000003.10.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000014.42.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:29.674 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-16 2020-12-05T19:12:29.675 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-16 zones={test-zone2, test-zone1} 2020-12-05T19:12:31.007 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:12:31.015 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:31.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:31.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:31.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:31.017 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:12:31.025 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:31.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:31.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:31.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:31.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:31.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:31.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:31.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:31.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:31.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:31.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:31.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:31.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:31.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:31.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:31.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:32.036 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-16 zones={test-zone2, test-zone1} 2020-12-05T19:12:32.037 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-16 zones={test-zone3, test-zone1} 2020-12-05T19:12:32.232 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:12:32.237 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:32.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:32.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:32.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:32.239 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:12:32.244 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:32.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:32.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:32.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:32.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:32.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:32.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:32.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:32.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:32.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:32.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:32.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:32.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:32.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:33.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:33.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:33.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-16 zones={test-zone3, test-zone1} 2020-12-05T19:12:33.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-16 zones={test-zone3, test-zone2} 2020-12-05T19:12:33.861 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:12:33.872 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:33.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:33.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:33.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:33.874 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:12:33.882 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:33.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:33.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:33.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:33.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:33.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:33.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:34.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:34.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:34.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:34.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:34.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:34.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:34.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:34.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:34.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:34.889 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-16 zones={test-zone3, test-zone2} 2020-12-05T19:12:34.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:34.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:34.891 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:34.934 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:34.934 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:35.163 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.38.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:35.163 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:12:35.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:35.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:35.165 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:35.468 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000010.38.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000007.28.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:35.469 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:12:35.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:35.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:35.471 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:35.513 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:35.513 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:35.737 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.38.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:35.737 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:12:35.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:35.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:35.739 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:35.822 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:35.823 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:36.079 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000010.38.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000007.28.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:36.079 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:12:36.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:36.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:36.081 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:36.163 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:36.163 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:36.404 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.38.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:36.405 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:12:36.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:36.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:36.408 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:36.454 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:36.454 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:36.729 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.38.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.28.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:36.729 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:12:36.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:36.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:36.731 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:36.813 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:36.813 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:37.039 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.24.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.30.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:37.039 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:12:37.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:37.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:37.041 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:37.085 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:37.086 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:37.332 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:37.332 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:12:37.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:37.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:37.334 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:37.374 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:37.375 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:37.596 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.24.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.30.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:37.597 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:12:37.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:37.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:37.598 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:37.645 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:37.645 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:37.889 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:37.890 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:12:37.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:37.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:37.892 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:37.938 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:37.938 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:38.166 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.23.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.19.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.43.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:38.167 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-17 2020-12-05T19:12:38.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:38.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:38.168 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:38.246 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:38.247 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:38.526 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000014.43.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:38.526 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-17 2020-12-05T19:12:38.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:38.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:38.528 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:38.605 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:38.606 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:38.841 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.23.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.19.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.43.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:38.842 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-17 2020-12-05T19:12:38.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:38.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:38.843 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:38.889 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:38.889 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:39.137 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000014.43.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:39.138 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-17 2020-12-05T19:12:39.138 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-17 zones={test-zone2, test-zone1} 2020-12-05T19:12:39.937 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:12:39.945 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:39.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:39.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:39.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:39.947 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:12:39.953 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:39.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:39.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:39.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:39.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:39.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:40.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:40.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:40.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:40.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:40.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:40.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:40.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:40.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:40.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:40.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:40.938 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-17 zones={test-zone2, test-zone1} 2020-12-05T19:12:40.939 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-17 zones={test-zone3, test-zone1} 2020-12-05T19:12:41.048 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:12:41.056 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:41.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:41.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:41.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:41.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:12:41.067 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:41.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:41.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:41.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:41.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:41.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:41.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:41.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:41.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:41.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:41.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:41.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:41.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:41.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:41.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:41.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:42.042 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-17 zones={test-zone3, test-zone1} 2020-12-05T19:12:42.043 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-17 zones={test-zone3, test-zone2} 2020-12-05T19:12:42.959 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:12:42.967 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:42.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:42.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:42.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:42.969 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:12:42.975 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:42.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:42.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:42.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:42.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:43.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:43.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:43.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:43.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:43.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:43.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:43.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:43.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:43.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:43.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:43.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:43.961 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-17 zones={test-zone3, test-zone2} 2020-12-05T19:12:43.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:43.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:43.963 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:44.004 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:44.005 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:44.240 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.34.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:44.241 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:12:44.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:44.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:44.242 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:44.286 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:44.286 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:44.554 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000010.25.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000007.34.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:44.555 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:12:44.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:44.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:44.556 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:44.597 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:44.597 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:44.823 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.34.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:44.824 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:12:44.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:44.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:44.826 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:44.906 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:44.907 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:45.154 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000010.25.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000007.34.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:45.155 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:12:45.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:45.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:45.158 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:45.238 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:45.238 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:45.455 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.34.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:45.456 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:12:45.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:45.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:45.457 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:45.502 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:45.502 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:45.778 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.25.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.34.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:45.778 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:12:45.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:45.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:45.780 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:45.865 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:46.115 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.8.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.28.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:46.115 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-18 2020-12-05T19:12:46.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:46.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:46.117 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:46.161 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:46.162 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:46.413 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000003.8.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000010.28.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000007.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:46.414 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-18 2020-12-05T19:12:46.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:46.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:46.415 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:46.457 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:46.458 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:46.698 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.8.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.28.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:46.699 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-18 2020-12-05T19:12:46.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:46.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:46.700 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:46.744 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:46.744 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:46.990 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000003.8.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000010.28.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000007.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:46.990 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-18 2020-12-05T19:12:46.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:46.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:46.992 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:47.072 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:47.072 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:47.312 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.10.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.18.9"},{"key":4,"val":""},{"key":5,"val":"00000000015.40.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:47.312 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:12:47.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:47.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:47.314 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:47.397 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:47.398 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:47.662 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000003.10.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000015.40.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:47.663 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:12:47.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:47.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:47.667 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:47.754 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:47.755 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:47.989 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.10.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.18.9"},{"key":4,"val":""},{"key":5,"val":"00000000015.40.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:47.990 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:12:47.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:47.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:47.992 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:48.046 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:48.326 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000003.10.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000015.40.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:48.327 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:12:48.327 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-18 zones={test-zone2, test-zone1} 2020-12-05T19:12:49.477 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:12:49.486 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:49.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:49.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:49.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:49.488 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:12:49.495 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:49.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:49.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:49.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:49.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:49.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:49.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:49.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:49.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:49.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:49.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:49.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:50.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:50.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:50.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:50.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:50.495 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-18 zones={test-zone2, test-zone1} 2020-12-05T19:12:50.496 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-18 zones={test-zone3, test-zone1} 2020-12-05T19:12:51.140 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:12:51.147 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:51.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:51.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:51.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:51.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:12:51.158 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:51.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:51.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:51.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:51.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:51.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:51.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:51.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:51.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:51.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:51.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:51.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:51.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:51.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:51.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:52.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:52.165 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-18 zones={test-zone3, test-zone1} 2020-12-05T19:12:52.166 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-18 zones={test-zone3, test-zone2} 2020-12-05T19:12:52.668 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:12:52.676 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:52.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:52.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:52.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:52.678 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:12:52.684 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:12:52.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:12:52.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:12:52.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:12:52.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:52.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:52.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:12:52.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:52.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:53.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:12:53.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:53.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:53.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:12:53.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:53.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:53.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:12:53.695 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-18 zones={test-zone3, test-zone2} 2020-12-05T19:12:54.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:54.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:54.760 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:54.797 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:54.797 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:55.032 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.43.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.38.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:55.032 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:55.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:55.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:55.034 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:55.077 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:55.078 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:55.326 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000008.38.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:55.326 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:55.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:55.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:55.328 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:55.370 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:55.370 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:55.603 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.43.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:12:55.604 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:12:55.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:55.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:12:55.605 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:12:55.691 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:12:55.692 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:12:55.953 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000008.38.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:12:55.954 WARNING:rgw_multi.tests:bucket tazcyr-16 zone test-zone3 behind zone test-zone1: shard=3 master=00000000004.19.5 target=00000000003.16.9, shard=5 master=00000000009.41.5 target=00000000008.38.9 2020-12-05T19:13:00.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:00.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:00.957 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:01.000 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:01.000 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:01.243 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000009.41.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:01.244 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:13:01.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:01.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:01.245 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:01.322 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:01.322 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:01.562 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.43.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:01.562 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:13:01.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:01.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:01.564 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:01.609 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:01.609 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:01.887 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.41.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:01.887 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-16 2020-12-05T19:13:01.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:01.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:01.889 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:01.931 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:01.932 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:02.155 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.22.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.32.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:02.155 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-16 2020-12-05T19:13:02.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:02.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:02.157 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:02.202 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:02.202 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:02.451 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000011.32.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000008.29.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:02.452 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-16 2020-12-05T19:13:02.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:02.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:02.453 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:02.496 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:02.497 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:02.737 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.22.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.32.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:02.737 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-16 2020-12-05T19:13:02.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:02.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:02.739 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:02.781 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:02.782 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:03.038 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000011.32.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000008.29.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:03.039 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-16 2020-12-05T19:13:03.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:03.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:03.040 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:03.122 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:03.123 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:03.342 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.21.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.13.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:03.343 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-16 2020-12-05T19:13:03.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:03.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:03.344 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:03.425 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:03.425 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:03.668 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000004.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000015.45.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:03.669 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-16 2020-12-05T19:13:03.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:03.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:03.671 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:03.753 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:03.754 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:03.974 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.21.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.13.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:03.974 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-16 2020-12-05T19:13:03.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:03.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:03.976 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:04.022 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:04.022 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:04.280 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ja8uF0InXTUVWIgdCxUy0qeNKcuZwil","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"wvH9zmRwkvMm4tjdnN92u-JSIErSY98","ns":""},"count":1},"inc_marker":{"position":"00000000004.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"lylO6VyGWB-puwvsl-6YbangTD6TKLi","ns":""},"count":1},"inc_marker":{"position":"00000000015.45.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:04.281 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-16 2020-12-05T19:13:04.281 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-16 zones={test-zone2, test-zone1} 2020-12-05T19:13:05.499 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:05.508 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:05.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:05.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:05.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:05.511 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:05.520 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:05.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:05.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:05.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:05.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:05.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:05.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:05.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:05.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:05.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:05.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:06.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:06.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:06.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:06.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:06.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:06.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-16 zones={test-zone2, test-zone1} 2020-12-05T19:13:06.536 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-16 zones={test-zone3, test-zone1} 2020-12-05T19:13:06.559 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:06.565 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:06.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:06.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:06.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:06.567 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:06.572 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:06.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:06.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:06.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:06.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:06.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:06.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:06.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:06.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:06.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:07.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:07.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:07.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:07.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:07.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:07.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:07.567 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-16 zones={test-zone3, test-zone1} 2020-12-05T19:13:07.568 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-16 zones={test-zone3, test-zone2} 2020-12-05T19:13:08.404 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:08.415 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:08.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:08.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:08.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:08.417 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:08.432 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:08.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:08.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:08.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:08.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:08.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:08.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:08.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:08.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:08.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:08.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:08.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:09.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:09.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:09.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:09.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:09.461 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-16 zones={test-zone3, test-zone2} 2020-12-05T19:13:09.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:09.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:09.463 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:09.504 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:09.504 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:09.736 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.45.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.31.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:09.736 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:13:09.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:09.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:09.738 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:09.777 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:09.777 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:10.032 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000011.45.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000008.31.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:10.033 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:13:10.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:10.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:10.034 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:10.076 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:10.076 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:10.291 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.45.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.31.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:10.292 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:13:10.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:10.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:10.293 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:10.373 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:10.374 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:10.619 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000011.45.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000008.31.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:10.620 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:13:10.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:10.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:10.621 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:10.698 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:10.698 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:10.927 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.45.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.31.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:10.928 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:13:10.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:10.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:10.929 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:10.973 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:10.974 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:11.353 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.45.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.31.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:11.353 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-17 2020-12-05T19:13:11.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:11.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:11.355 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:11.395 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:11.395 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:11.634 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.27.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.31.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:11.634 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:13:11.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:11.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:11.636 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:11.680 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:11.680 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:11.924 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000004.27.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000011.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000010.31.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:11.926 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:13:11.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:11.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:11.927 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:11.967 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:11.968 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:12.222 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.27.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.31.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:12.223 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:13:12.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:12.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:12.225 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:12.270 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:12.271 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:12.518 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000004.27.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000011.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000010.31.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:12.519 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-17 2020-12-05T19:13:12.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:12.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:12.521 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:12.600 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:12.601 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:12.853 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.26.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.22.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.46.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:12.855 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-17 2020-12-05T19:13:12.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:12.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:12.860 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:12.947 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:12.948 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:13.201 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000015.46.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:13.201 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-17 2020-12-05T19:13:13.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:13.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:13.203 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:13.300 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:13.300 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:13.529 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.26.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.22.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.46.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:13.530 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-17 2020-12-05T19:13:13.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:13.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:13.536 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:13.586 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:13.586 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:13.834 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"y61Wnw6LUol-trW.-MNJ78EpPv8CqIk","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"qWfai8mpZ.7Seqc4GZUQUbNSOiD3DLn","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TnqCp4lOBiZUVYdWrKexadnjxZFX-CT","ns":""},"count":1},"inc_marker":{"position":"00000000015.46.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:13.835 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-17 2020-12-05T19:13:13.836 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-17 zones={test-zone2, test-zone1} 2020-12-05T19:13:15.369 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:15.378 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:15.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:15.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:15.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:15.381 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:15.390 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:15.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:15.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:15.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:15.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:15.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:15.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:15.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:15.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:16.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:16.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:16.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:16.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:16.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:16.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:16.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:16.759 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-17 zones={test-zone2, test-zone1} 2020-12-05T19:13:16.759 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-17 zones={test-zone3, test-zone1} 2020-12-05T19:13:16.842 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:16.848 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:16.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:16.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:16.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:16.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:16.856 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:16.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:16.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:16.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:16.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:16.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:16.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:17.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:17.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:17.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:17.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:17.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:17.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:17.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:17.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:17.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:17.858 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-17 zones={test-zone3, test-zone1} 2020-12-05T19:13:17.859 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-17 zones={test-zone3, test-zone2} 2020-12-05T19:13:18.486 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:18.495 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:18.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:18.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:18.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:18.497 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:18.504 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:18.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:18.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:18.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:18.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:18.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:18.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:18.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:18.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:18.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:18.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:18.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:19.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:19.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:19.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:19.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:19.497 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-17 zones={test-zone3, test-zone2} 2020-12-05T19:13:19.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:19.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:19.499 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:19.541 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:19.541 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:19.777 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:19.777 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:13:19.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:19.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:19.779 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:19.822 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:19.822 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:20.078 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000011.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000008.37.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:20.079 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:13:20.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:20.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:20.081 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:20.123 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:20.123 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:20.341 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:20.342 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:13:20.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:20.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:20.343 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:20.428 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:20.428 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:20.691 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000011.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000008.37.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:20.692 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:13:20.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:20.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:20.693 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:20.773 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:20.773 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:20.994 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:20.995 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:13:20.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:20.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:20.997 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:21.042 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:21.042 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:21.324 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.37.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:21.324 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-18 2020-12-05T19:13:21.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:21.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:21.326 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:21.370 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:21.641 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.11.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.27.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:21.641 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-18 2020-12-05T19:13:21.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:21.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:21.643 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:21.687 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:21.688 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:21.920 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000004.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000011.34.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000008.27.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:21.921 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-18 2020-12-05T19:13:21.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:21.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:21.923 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:21.965 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:21.965 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:22.200 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.11.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.27.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:22.200 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-18 2020-12-05T19:13:22.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:22.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:22.202 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:22.250 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:22.250 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:22.495 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000004.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000011.34.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000008.27.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:22.495 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-18 2020-12-05T19:13:22.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:22.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:22.497 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:22.536 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:22.537 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:22.757 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.21.5"},{"key":4,"val":""},{"key":5,"val":"00000000016.43.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:22.758 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:13:22.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:22.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:22.760 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:22.842 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:22.842 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:23.088 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000004.13.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000016.43.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:23.089 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:13:23.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:23.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:23.090 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:23.171 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:23.172 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:23.400 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.21.5"},{"key":4,"val":""},{"key":5,"val":"00000000016.43.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:23.400 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:13:23.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:23.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:23.402 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:23.447 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:23.448 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:23.704 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"tguOoxVsgE0u8aQja6UnWUBs5VXlIHt","ns":""},"count":1},"inc_marker":{"position":"00000000004.13.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"NWBuUkP4f0wDqMJjy3OAYVzxXwAPHAm","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"jvWa5AnLcJ8HZYNLYtziBC8QLfKI364","ns":""},"count":1},"inc_marker":{"position":"00000000016.43.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:23.705 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-18 2020-12-05T19:13:23.705 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-18 zones={test-zone2, test-zone1} 2020-12-05T19:13:24.810 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:24.819 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:24.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:24.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:24.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:24.821 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:24.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:24.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:24.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:24.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:24.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:24.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:24.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:25.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:25.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:25.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:25.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:25.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:25.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:25.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:25.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:25.847 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-18 zones={test-zone2, test-zone1} 2020-12-05T19:13:25.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-18 zones={test-zone3, test-zone1} 2020-12-05T19:13:26.311 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:26.316 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:26.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:26.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:26.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:26.318 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:26.323 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:26.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:26.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:26.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:26.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:26.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:26.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:26.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:26.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:26.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:26.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:26.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:26.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:27.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:27.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:27.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:27.319 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-18 zones={test-zone3, test-zone1} 2020-12-05T19:13:27.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-18 zones={test-zone3, test-zone2} 2020-12-05T19:13:28.050 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:28.060 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:28.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:28.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:28.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:28.062 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:28.068 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:28.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-12-05T19:13:28.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-12-05T19:13:28.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-12-05T19:13:28.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:28.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:28.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:28.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:28.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:28.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-12-05T19:13:28.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:28.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:28.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-12-05T19:13:28.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:28.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:28.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-12-05T19:13:29.029 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-18 zones={test-zone3, test-zone2} 2020-12-05T19:13:29.030 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-12-05T19:13:29.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:29.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:29.488 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:29.526 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:29.526 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:29.723 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:13:29.756 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195609.483992_208.1","last_update":"2020-12-05T19:13:29.483992Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:13:29.757 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:13:29.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:29.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:29.759 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:29.794 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:29.794 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:30.031 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195504.311460_197.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:11:44.311460Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:13:30.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195609.483992_208.1 target=1_1607195504.311460_197.1 2020-12-05T19:13:35.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:35.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:35.034 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:35.076 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:35.328 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195609.483992_208.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:29.483992Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:13:35.329 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:13:35.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:13:35.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:35.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:35.331 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:35.376 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:35.377 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:35.611 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195609.483992_208.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:29.483992Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:13:35.611 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:13:35.612 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:13:35.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:35.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:35.613 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:35.934 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195609.483992_208.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:29.483992Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:13:35.934 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:13:36.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:36.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:36.430 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:36.474 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:36.474 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:36.713 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:36.713 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:36.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:36.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:36.715 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:36.765 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:36.765 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:37.010 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:37.011 WARNING:rgw_multi.tests:bucket tazcyr-19 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.14.10 target= 2020-12-05T19:13:42.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:42.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:42.014 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:42.059 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:42.060 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:42.302 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:42.303 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:42.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:42.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:42.305 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:42.346 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:42.347 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:42.578 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:42.579 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:42.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:42.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:42.580 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:42.666 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:42.666 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:42.910 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:42.911 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:42.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:42.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:42.913 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:42.952 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:42.952 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:43.183 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:43.184 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:43.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:43.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:43.185 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:43.229 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:43.230 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:43.497 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:43.498 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:43.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:43.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:43.500 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:43.589 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:43.590 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:43.798 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:43.799 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-19 2020-12-05T19:13:43.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:43.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:43.800 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:43.845 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:43.845 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:44.118 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:44.118 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-19 2020-12-05T19:13:44.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:44.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:44.120 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:44.162 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:44.163 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:44.397 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:44.398 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-19 2020-12-05T19:13:44.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:44.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:44.400 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:44.443 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:44.444 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:44.697 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:44.698 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-19 2020-12-05T19:13:44.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:44.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:44.699 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:44.785 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:44.786 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:45.013 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:45.013 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-19 2020-12-05T19:13:45.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:45.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:45.015 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:45.101 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:45.101 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:45.348 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:45.349 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-19 2020-12-05T19:13:45.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:45.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:45.350 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:45.432 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:45.432 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:45.658 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:45.659 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-19 2020-12-05T19:13:45.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:45.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:45.660 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:45.705 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:45.705 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:45.949 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:45.950 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-19 2020-12-05T19:13:45.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-19 zones={test-zone2, test-zone1} 2020-12-05T19:13:47.191 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:47.198 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:47.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:47.246 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:47.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:47.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:47.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:47.462 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-19 zones={test-zone2, test-zone1} 2020-12-05T19:13:47.462 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-19 zones={test-zone3, test-zone1} 2020-12-05T19:13:47.548 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:47.594 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:47.595 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:47.603 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:47.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:47.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:47.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:47.841 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-19 zones={test-zone3, test-zone1} 2020-12-05T19:13:47.841 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-19 zones={test-zone3, test-zone2} 2020-12-05T19:13:47.851 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:13:47.902 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:47.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:13:47.911 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:13:47.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:47.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:48.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:13:48.109 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-19 zones={test-zone3, test-zone2} 2020-12-05T19:13:49.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:49.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:49.815 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:49.852 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:49.853 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:50.069 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000038.132.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:50.070 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:50.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:50.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:50.072 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:50.116 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:50.116 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:50.345 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"xa8IDdq1LDKpO2uG-C1-uTkrkSVGIrS","ns":""},"count":6},"inc_marker":{"position":"00000000039.135.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:50.346 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:50.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:50.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:50.347 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:50.392 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:50.393 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:50.614 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000044.146.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:50.615 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:50.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:50.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:50.616 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:50.700 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:50.701 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:50.946 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"xa8IDdq1LDKpO2uG-C1-uTkrkSVGIrS","ns":""},"count":6},"inc_marker":{"position":"00000000044.146.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:50.946 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:50.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:50.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:50.948 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:51.030 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:51.030 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:51.272 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000044.146.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:51.273 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:51.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:51.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:51.274 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:51.313 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:51.314 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:51.584 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000044.146.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:51.585 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-19 2020-12-05T19:13:51.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:51.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:51.586 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:51.666 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:51.666 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:51.909 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.142.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:51.910 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-19 2020-12-05T19:13:51.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:51.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:51.912 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:51.956 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:51.956 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:52.215 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nWOS9Q2MydXFgLHXb7005DsP.keovMn","ns":""},"count":5},"inc_marker":{"position":"00000000046.140.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:52.216 WARNING:rgw_multi.tests:bucket tazcyr-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000047.142.9 target=00000000046.140.9 2020-12-05T19:13:57.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:57.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:57.218 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:57.262 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:57.263 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:57.519 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nWOS9Q2MydXFgLHXb7005DsP.keovMn","ns":""},"count":5},"inc_marker":{"position":"00000000047.142.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:57.519 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-19 2020-12-05T19:13:57.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:57.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:57.521 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:57.567 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:57.567 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:57.765 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.142.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:57.765 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-19 2020-12-05T19:13:57.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:57.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:57.767 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:57.811 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:57.812 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:58.062 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nWOS9Q2MydXFgLHXb7005DsP.keovMn","ns":""},"count":5},"inc_marker":{"position":"00000000047.142.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:58.062 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-19 2020-12-05T19:13:58.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:58.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:58.064 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:58.168 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:58.168 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:58.383 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000050.140.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:58.384 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-19 2020-12-05T19:13:58.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:58.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:58.385 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:58.474 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:58.474 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:58.742 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nWOS9Q2MydXFgLHXb7005DsP.keovMn","ns":""},"count":9},"inc_marker":{"position":"00000000050.140.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:58.743 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-19 2020-12-05T19:13:58.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:58.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:58.744 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:58.823 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:58.824 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:59.052 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000050.140.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:13:59.053 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-19 2020-12-05T19:13:59.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:59.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:13:59.054 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:13:59.100 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:13:59.101 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:13:59.356 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"bnhHA9hP703KrpdlqBymMaov.Ya9xgt","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nWOS9Q2MydXFgLHXb7005DsP.keovMn","ns":""},"count":9},"inc_marker":{"position":"00000000050.140.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:13:59.357 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-19 2020-12-05T19:13:59.357 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-19 zones={test-zone2, test-zone1} 2020-12-05T19:14:00.140 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:14:00.150 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:14:00.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.162 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:14:00.173 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:14:00.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:00.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:14:00.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:14:00.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:14:00.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:00.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:00.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:00.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:00.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:00.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:00.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:01.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:02.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:03.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:03.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:03.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:03.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:03.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:03.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:03.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:03.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:03.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:04.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:04.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:04.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:04.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:04.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:04.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:04.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:04.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:04.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:05.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:06.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:07.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:08.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:09.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:09.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:09.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:09.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:09.436 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-19 zones={test-zone2, test-zone1} 2020-12-05T19:14:09.437 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-19 zones={test-zone3, test-zone1} 2020-12-05T19:14:09.685 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:14:09.692 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:14:09.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.706 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:14:09.715 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:14:09.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:09.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:14:09.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:14:09.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:14:09.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:09.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:10.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:11.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:12.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:13.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:14.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:15.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:16.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:17.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:17.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:17.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:17.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:17.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:17.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:17.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:17.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:17.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:18.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:18.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:18.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:18.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:18.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:18.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:18.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:18.747 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-19 zones={test-zone3, test-zone1} 2020-12-05T19:14:18.748 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-19 zones={test-zone3, test-zone2} 2020-12-05T19:14:18.877 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:14:18.887 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:14:18.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.900 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:14:18.911 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-12-05T19:14:18.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:18.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:14:18.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:14:19.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-12-05T19:14:19.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:19.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:19.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:19.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:19.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:19.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:19.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:19.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:19.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:19.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:20.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:21.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:21.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:21.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:21.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:21.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:21.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:21.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:21.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:21.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:22.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:22.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:22.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:22.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:22.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:22.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:22.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:22.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:22.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:23.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:24.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:25.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:26.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:26.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:26.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:26.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:26.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:26.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:26.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:26.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:27.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:27.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:27.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:27.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:27.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:27.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:27.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:27.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:27.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:28.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:28.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:28.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:28.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:28.522 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-19 zones={test-zone3, test-zone2} 2020-12-05T19:14:28.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:28.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:28.524 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:28.561 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:28.562 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:28.791 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195646.112070_77.1","last_update":"2020-12-05T19:14:06.112070Z"},{"marker":"1_1607195624.108585_165.1","last_update":"2020-12-05T19:13:44.108585Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1607195580.099522_101.3","last_update":"2020-12-05T19:13:00.099522Z"}] 2020-12-05T19:14:28.792 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:14:28.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:28.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:28.795 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:28.839 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:28.840 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:29.078 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195646.112070_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:06.112070Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.108585_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.108585Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:14:29.079 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:14:29.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:29.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:29.081 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:29.122 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:29.123 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:29.352 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195646.112070_77.1","last_update":"2020-12-05T19:14:06.112070Z"},{"marker":"1_1607195624.108585_165.1","last_update":"2020-12-05T19:13:44.108585Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1607195580.099522_101.3","last_update":"2020-12-05T19:13:00.099522Z"}] 2020-12-05T19:14:29.352 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:14:29.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:29.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:29.354 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:29.434 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:29.435 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:29.670 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10689636149165243743},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195646.112070_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:06.112070Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.108585_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.108585Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:14:29.671 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:14:29.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:29.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:29.672 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:29.749 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:29.750 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:29.970 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195646.112070_77.1","last_update":"2020-12-05T19:14:06.112070Z"},{"marker":"1_1607195624.108585_165.1","last_update":"2020-12-05T19:13:44.108585Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1607195580.099522_101.3","last_update":"2020-12-05T19:13:00.099522Z"}] 2020-12-05T19:14:29.970 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:14:29.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:29.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:29.972 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:30.013 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:30.014 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:30.290 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195646.112070_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:06.112070Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.108585_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.108585Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:14:30.291 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:14:30.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:30.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:30.292 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:30.379 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:30.380 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:30.617 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607195647.247106_68.1","last_update":"2020-12-05T19:14:07.247106Z"},{"marker":"1_1607195625.242812_162.1","last_update":"2020-12-05T19:13:45.242812Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1607195581.234035_126.3","last_update":"2020-12-05T19:13:01.234035Z"}] 2020-12-05T19:14:30.618 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:14:30.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:30.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:30.620 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:30.661 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:30.662 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:30.902 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9760760649708652305},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195647.247106_68.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:07.247106Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195625.242812_162.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:45.242812Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195581.234035_126.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:01.234035Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:14:30.903 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:14:30.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:30.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:30.905 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:30.944 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:30.945 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:31.170 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607195647.247106_68.1","last_update":"2020-12-05T19:14:07.247106Z"},{"marker":"1_1607195625.242812_162.1","last_update":"2020-12-05T19:13:45.242812Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1607195581.234035_126.3","last_update":"2020-12-05T19:13:01.234035Z"}] 2020-12-05T19:14:31.171 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:14:31.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:31.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:31.173 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:31.217 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:31.217 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:31.444 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3761623777010785199},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195647.247106_68.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:07.247106Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195625.242812_162.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:45.242812Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195581.234035_126.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:01.234035Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:14:31.444 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:14:31.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:31.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:31.447 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:31.522 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:31.522 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:31.757 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195646.114056_55.1","last_update":"2020-12-05T19:14:06.114056Z"},{"marker":"1_1607195624.110351_128.1","last_update":"2020-12-05T19:13:44.110351Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1607195580.100307_98.3","last_update":"2020-12-05T19:13:00.100307Z"}] 2020-12-05T19:14:31.758 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:14:31.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:31.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:31.759 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:31.848 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:31.849 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:32.107 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4132799784717954996},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195646.114056_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:06.114056Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.110351_128.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.110351Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.100307_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.100307Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:14:32.108 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:14:32.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:32.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:32.109 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:32.188 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:32.188 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:32.420 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195646.114056_55.1","last_update":"2020-12-05T19:14:06.114056Z"},{"marker":"1_1607195624.110351_128.1","last_update":"2020-12-05T19:13:44.110351Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1607195580.100307_98.3","last_update":"2020-12-05T19:13:00.100307Z"}] 2020-12-05T19:14:32.420 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:14:32.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:32.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:32.422 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:32.465 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:32.465 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:32.695 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6833204601607133234},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195646.114056_55.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:06.114056Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.110351_128.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.110351Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.100307_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.100307Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:14:32.695 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:14:32.696 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-12-05T19:14:33.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:33.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:33.677 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:33.713 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:33.713 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:33.933 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:14:33.963 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195673.670005_213.1","last_update":"2020-12-05T19:14:33.670005Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:14:33.963 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:14:33.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:33.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:33.965 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:34.004 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:34.004 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:34.247 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195609.483992_208.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:29.483992Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:14:34.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195673.670005_213.1 target=1_1607195609.483992_208.1 2020-12-05T19:14:39.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:39.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:39.249 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:39.291 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:39.291 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:39.532 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195673.670005_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:33.670005Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:14:39.532 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:14:39.533 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:14:39.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:39.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:39.534 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:39.575 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:39.575 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:39.812 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195673.670005_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:33.670005Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:14:39.813 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:14:39.813 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:14:39.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:39.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:39.815 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:39.854 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:40.128 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195673.670005_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:33.670005Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:14:40.129 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:14:40.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:40.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:40.264 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:40.304 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:40.304 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:40.543 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:14:40.543 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:14:40.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:40.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:40.545 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:40.588 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:40.588 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:40.851 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:14:40.852 WARNING:rgw_multi.tests:bucket tazcyr-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000001.12.5 target= 2020-12-05T19:14:45.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:45.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:45.854 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:46.153 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:14:46.154 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:14:46.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:46.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:46.156 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:46.197 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:46.197 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:46.422 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:14:46.423 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:14:46.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:46.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:46.424 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:46.509 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:46.509 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:46.764 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:14:46.765 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:14:46.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:46.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:46.767 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:46.809 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:46.809 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:47.036 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:14:47.036 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:14:47.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:47.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:47.038 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:47.083 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:47.083 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:47.354 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.12.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:14:47.354 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:14:47.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:47.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:47.356 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:47.399 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:47.399 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:47.624 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:14:47.624 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:14:47.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:47.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:47.627 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:47.672 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:47.672 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:47.936 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:14:47.936 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:14:47.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:47.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:47.938 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:47.980 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:47.980 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:48.220 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:14:48.221 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:14:48.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:48.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:48.224 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:48.271 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:48.271 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:48.515 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:14:48.515 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:14:48.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:48.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:48.517 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:48.559 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:48.559 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:49.164 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:14:49.164 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:14:49.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:49.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:49.166 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:49.248 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:49.248 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:49.502 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:14:49.503 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:14:49.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:49.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:49.505 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:49.586 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:49.586 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:49.823 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:14:49.823 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:14:49.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:49.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:49.825 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:49.872 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:49.872 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:50.138 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:14:50.139 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:14:50.139 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-20 zones={test-zone2, test-zone1} 2020-12-05T19:14:50.751 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:14:50.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:50.758 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:14:50.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:50.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:50.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:50.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:51.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-20 zones={test-zone2, test-zone1} 2020-12-05T19:14:51.019 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-20 zones={test-zone3, test-zone1} 2020-12-05T19:14:52.022 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:14:52.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:52.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:14:52.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:52.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:52.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:52.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:52.287 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-20 zones={test-zone3, test-zone1} 2020-12-05T19:14:52.288 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-20 zones={test-zone3, test-zone2} 2020-12-05T19:14:52.713 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:14:52.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:52.794 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:14:52.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:14:52.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:52.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:52.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:14:53.004 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-20 zones={test-zone3, test-zone2} 2020-12-05T19:14:53.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:53.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:53.023 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:53.059 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:53.060 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:53.261 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:14:53.297 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195693.016790_215.1","last_update":"2020-12-05T19:14:53.016790Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:14:53.298 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:14:53.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:53.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:53.300 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:53.340 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:53.340 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:53.579 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195673.670005_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:33.670005Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:14:53.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195693.016790_215.1 target=1_1607195673.670005_213.1 2020-12-05T19:14:58.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:58.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:58.582 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:58.623 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:58.623 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:58.873 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195693.016790_215.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:53.016790Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:14:58.874 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:14:58.875 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:14:58.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:58.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:58.877 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:58.920 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:58.920 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:59.155 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195693.016790_215.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:53.016790Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:14:59.155 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:14:59.156 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:14:59.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:59.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:59.157 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:59.199 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:59.199 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:59.477 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195693.016790_215.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:53.016790Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:14:59.477 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:14:59.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:59.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:59.732 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:14:59.772 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:14:59.773 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:14:59.998 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.15.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:14:59.998 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:14:59.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:14:59.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:00.000 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:00.046 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:00.047 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:00.298 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:00.299 WARNING:rgw_multi.tests:bucket tazcyr-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.15.12 target=00000000001.12.5 2020-12-05T19:15:05.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:05.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:05.302 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:05.346 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:05.346 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:05.601 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:05.602 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:05.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:05.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:05.603 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:05.643 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:05.644 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:05.880 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.15.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:05.881 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:05.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:05.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:05.882 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:05.966 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:05.966 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:06.209 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:06.210 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:06.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:06.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:06.211 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:06.248 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:06.249 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:06.482 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.15.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:06.482 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:06.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:06.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:06.484 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:06.528 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:06.528 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:06.807 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.15.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:06.807 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:06.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:06.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:06.809 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:06.891 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:06.891 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:07.129 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.24.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:07.130 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:15:07.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:07.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:07.132 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:07.175 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:07.175 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:07.435 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:07.435 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:15:07.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:07.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:07.437 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:07.481 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:07.481 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:07.722 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.24.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:07.723 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:15:07.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:07.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:07.724 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:07.769 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:07.769 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:08.016 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:08.016 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:15:08.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:08.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:08.018 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:08.098 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:08.098 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:08.350 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.23.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:08.351 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:15:08.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:08.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:08.353 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:08.450 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:08.450 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:08.715 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:08.716 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:15:08.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:08.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:08.718 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:08.796 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:08.797 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:09.033 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.23.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:09.033 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:15:09.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:09.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:09.035 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:09.348 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:09.349 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:15:09.349 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-20 zones={test-zone2, test-zone1} 2020-12-05T19:15:11.346 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:11.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:11.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:11.736 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:11.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:11.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:11.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:11.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:11.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:11.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:11.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:12.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:12.091 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-20 zones={test-zone2, test-zone1} 2020-12-05T19:15:12.092 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-20 zones={test-zone3, test-zone1} 2020-12-05T19:15:12.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:12.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:12.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:12.567 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:12.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:12.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:12.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:12.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:12.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:12.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:12.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:12.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:12.930 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-20 zones={test-zone3, test-zone1} 2020-12-05T19:15:12.930 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-20 zones={test-zone3, test-zone2} 2020-12-05T19:15:12.941 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:12.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:12.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:12.995 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:13.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:13.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:13.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:13.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:13.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:13.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:13.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:13.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:13.361 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-20 zones={test-zone3, test-zone2} 2020-12-05T19:15:13.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:13.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:13.378 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:13.417 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:13.417 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:13.622 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:15:13.658 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195713.373474_217.1","last_update":"2020-12-05T19:15:13.373474Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:15:13.658 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:15:13.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:13.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:13.660 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:13.700 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:13.700 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:13.938 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195693.016790_215.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:14:53.016790Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:15:13.939 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195713.373474_217.1 target=1_1607195693.016790_215.1 2020-12-05T19:15:18.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:18.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:18.941 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:19.225 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195713.373474_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:13.373474Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:15:19.225 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:15:19.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:15:19.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:19.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:19.227 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:19.269 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:19.270 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:19.509 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195713.373474_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:13.373474Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:15:19.510 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:15:19.510 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:15:19.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:19.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:19.512 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:19.555 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:19.556 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:19.809 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195713.373474_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:13.373474Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:15:19.809 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:15:20.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:20.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:20.700 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:20.742 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:20.742 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:20.982 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.19.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:20.983 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:20.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:20.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:20.985 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:21.031 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:21.031 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:21.295 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.19.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:21.296 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:21.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:21.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:21.297 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:21.339 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:21.339 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:21.589 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.19.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:21.589 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:21.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:21.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:21.591 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:21.671 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:21.671 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:21.941 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.19.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:21.942 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:21.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:21.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:21.943 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:22.025 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:22.026 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:22.256 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.19.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:22.257 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:22.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:22.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:22.258 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:22.302 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:22.303 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:22.579 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.19.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:22.580 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-20 2020-12-05T19:15:22.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:22.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:22.582 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:22.663 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:22.664 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:22.913 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.32.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:22.914 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:15:22.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:22.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:22.916 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:22.962 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:22.962 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:23.221 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.32.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:23.221 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:15:23.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:23.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:23.223 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:23.264 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:23.265 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:23.524 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.32.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:23.525 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:15:23.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:23.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:23.526 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:23.573 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:23.574 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:23.832 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.32.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:23.833 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-20 2020-12-05T19:15:23.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:23.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:23.835 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:23.874 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:23.906 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:24.111 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.31.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:24.111 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:15:24.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:24.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:24.113 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:24.194 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:24.195 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:24.466 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.31.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:24.467 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:15:24.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:24.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:24.468 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:24.550 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:24.550 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:24.796 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.31.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:24.796 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:15:24.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:24.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:24.798 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:24.841 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:24.841 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:25.105 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.31.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:25.106 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-20 2020-12-05T19:15:25.107 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-20 zones={test-zone2, test-zone1} 2020-12-05T19:15:26.028 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:26.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:26.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:26.037 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:26.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:26.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:26.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:26.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:26.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:26.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:26.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:26.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:26.453 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-20 zones={test-zone2, test-zone1} 2020-12-05T19:15:26.453 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-20 zones={test-zone3, test-zone1} 2020-12-05T19:15:26.939 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:26.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:26.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:26.991 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:27.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:27.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:27.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.342 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-20 zones={test-zone3, test-zone1} 2020-12-05T19:15:27.343 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-20 zones={test-zone3, test-zone2} 2020-12-05T19:15:27.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:27.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:27.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:27.403 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:27.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:27.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:27.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:27.748 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-20 zones={test-zone3, test-zone2} 2020-12-05T19:15:27.749 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-12-05T19:15:27.749 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-21 2020-12-05T19:15:27.784 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-22 2020-12-05T19:15:27.849 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-23 2020-12-05T19:15:27.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:27.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:27.958 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:27.996 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:27.997 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:28.199 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:15:28.237 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195727.946934_235.1","last_update":"2020-12-05T19:15:27.946934Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:15:28.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:15:28.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:28.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:28.239 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:28.279 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:28.279 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:28.536 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195713.373474_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:13.373474Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:15:28.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195727.946934_235.1 target=1_1607195713.373474_217.1 2020-12-05T19:15:33.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:33.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:33.540 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:33.577 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:33.790 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195727.946934_235.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:27.946934Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:15:33.790 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:15:33.791 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:15:33.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:33.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:33.792 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:33.831 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:33.832 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:34.060 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195727.946934_235.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:27.946934Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:15:34.061 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:15:34.061 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:15:34.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:34.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:34.063 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:34.103 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:34.379 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195727.946934_235.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:27.946934Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:15:34.379 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:15:37.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:37.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:37.098 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:37.136 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:37.137 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:37.376 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:37.377 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-21 2020-12-05T19:15:37.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:37.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:37.378 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:37.422 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:37.422 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:37.664 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yh8Eg2ah12iT9PdEgwHBtJlsl7BUFqC","ns":""},"count":2},"inc_marker":{"position":"00000000003.17.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:37.664 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-21 2020-12-05T19:15:37.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:37.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:37.666 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:37.710 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:37.710 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:37.942 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:37.942 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-21 2020-12-05T19:15:37.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:37.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:37.944 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:37.984 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:37.985 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:38.249 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yh8Eg2ah12iT9PdEgwHBtJlsl7BUFqC","ns":""},"count":2},"inc_marker":{"position":"00000000003.17.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:38.249 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-21 2020-12-05T19:15:38.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:38.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:38.251 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:38.322 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:38.323 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:38.539 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:38.539 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-21 2020-12-05T19:15:38.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:38.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:38.541 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:38.587 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:38.588 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:38.845 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.17.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:38.845 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-21 2020-12-05T19:15:38.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:38.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:38.847 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:38.928 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:38.928 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:39.169 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.23.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:39.170 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-21 2020-12-05T19:15:39.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:39.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:39.172 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:39.215 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:39.216 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:39.496 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yh8Eg2ah12iT9PdEgwHBtJlsl7BUFqC","ns":""},"count":2},"inc_marker":{"position":"00000000002.23.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:39.497 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-21 2020-12-05T19:15:39.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:39.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:39.499 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:39.541 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:39.541 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:39.762 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.23.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:39.763 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-21 2020-12-05T19:15:39.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:39.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:39.764 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:39.809 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:39.810 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:40.055 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yh8Eg2ah12iT9PdEgwHBtJlsl7BUFqC","ns":""},"count":2},"inc_marker":{"position":"00000000002.23.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:40.056 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-21 2020-12-05T19:15:40.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:40.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:40.057 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:40.095 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:40.095 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:40.320 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:40.321 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-21 2020-12-05T19:15:40.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:40.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:40.322 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:40.404 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:40.405 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:40.645 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yh8Eg2ah12iT9PdEgwHBtJlsl7BUFqC","ns":""},"count":2},"inc_marker":{"position":"00000000002.21.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:40.645 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-21 2020-12-05T19:15:40.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:40.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:40.647 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:40.725 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:40.725 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:40.945 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:40.945 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-21 2020-12-05T19:15:40.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:40.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:40.947 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:40.991 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:40.991 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:41.248 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yh8Eg2ah12iT9PdEgwHBtJlsl7BUFqC","ns":""},"count":2},"inc_marker":{"position":"00000000002.21.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:41.249 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-21 2020-12-05T19:15:41.249 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-21 zones={test-zone2, test-zone1} 2020-12-05T19:15:42.822 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:42.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:42.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:42.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:42.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:42.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:42.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:42.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:42.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:43.026 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-21 zones={test-zone2, test-zone1} 2020-12-05T19:15:43.027 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-21 zones={test-zone3, test-zone1} 2020-12-05T19:15:43.738 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:43.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:43.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:43.791 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:43.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:43.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:43.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:43.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:43.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:43.948 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-21 zones={test-zone3, test-zone1} 2020-12-05T19:15:43.948 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-21 zones={test-zone3, test-zone2} 2020-12-05T19:15:43.958 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:44.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:44.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:44.012 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:44.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:44.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:44.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:44.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:44.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:44.169 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-21 zones={test-zone3, test-zone2} 2020-12-05T19:15:44.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:44.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:44.171 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:44.212 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:44.212 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:44.458 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.27.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:44.459 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-22 2020-12-05T19:15:44.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:44.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:44.460 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:44.503 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:44.503 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"n5-JNpZK9PhjdKra3xYgYfkMc7izBHk","ns":""},"count":2},"inc_marker":{"position":"00000000002.27.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:44.763 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-22 2020-12-05T19:15:44.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:44.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:44.765 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:44.806 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:44.807 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:45.058 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.27.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:45.059 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-22 2020-12-05T19:15:45.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:45.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:45.060 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:45.143 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:45.143 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:45.399 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"n5-JNpZK9PhjdKra3xYgYfkMc7izBHk","ns":""},"count":2},"inc_marker":{"position":"00000000002.27.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:45.399 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-22 2020-12-05T19:15:45.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:45.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:45.401 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:45.483 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:45.484 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:45.731 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.27.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:45.731 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-22 2020-12-05T19:15:45.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:45.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:45.733 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:45.779 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:45.779 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:46.155 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:46.155 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-22 2020-12-05T19:15:46.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:46.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:46.157 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:46.195 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:46.195 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:46.436 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.24.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:46.437 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-22 2020-12-05T19:15:46.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:46.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:46.438 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:46.483 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:46.484 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:46.758 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"n5-JNpZK9PhjdKra3xYgYfkMc7izBHk","ns":""},"count":2},"inc_marker":{"position":"00000000003.24.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:46.759 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-22 2020-12-05T19:15:46.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:46.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:46.760 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:46.803 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:46.803 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:47.056 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.24.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:47.056 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-22 2020-12-05T19:15:47.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:47.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:47.059 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:47.104 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:47.104 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:47.378 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"n5-JNpZK9PhjdKra3xYgYfkMc7izBHk","ns":""},"count":2},"inc_marker":{"position":"00000000003.24.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:47.379 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-22 2020-12-05T19:15:47.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:47.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:47.382 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:47.463 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:47.463 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:47.694 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.20.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:47.695 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-22 2020-12-05T19:15:47.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:47.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:47.696 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:47.779 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:47.779 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:48.055 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"n5-JNpZK9PhjdKra3xYgYfkMc7izBHk","ns":""},"count":2},"inc_marker":{"position":"00000000002.20.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:48.055 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-22 2020-12-05T19:15:48.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:48.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:48.057 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:48.136 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:48.136 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:48.385 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.20.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:48.385 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-22 2020-12-05T19:15:48.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:48.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:48.387 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:48.435 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:48.435 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:48.694 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"n5-JNpZK9PhjdKra3xYgYfkMc7izBHk","ns":""},"count":2},"inc_marker":{"position":"00000000002.20.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:48.695 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-22 2020-12-05T19:15:48.696 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-22 zones={test-zone2, test-zone1} 2020-12-05T19:15:49.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:49.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:49.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:49.391 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:49.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:49.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:49.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:49.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:49.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:49.661 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-22 zones={test-zone2, test-zone1} 2020-12-05T19:15:49.662 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-22 zones={test-zone3, test-zone1} 2020-12-05T19:15:50.096 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:50.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:50.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:50.148 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:50.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:50.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:50.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:50.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:50.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:50.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-22 zones={test-zone3, test-zone1} 2020-12-05T19:15:50.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-22 zones={test-zone3, test-zone2} 2020-12-05T19:15:50.313 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:50.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:50.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:50.363 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:50.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:50.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:50.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:50.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:50.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:50.521 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-22 zones={test-zone3, test-zone2} 2020-12-05T19:15:50.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:50.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:50.523 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:50.565 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:50.565 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:50.809 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:50.809 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-23 2020-12-05T19:15:50.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:50.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:50.811 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:50.855 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:50.855 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:51.122 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"U5IsFzxpWezYRYGUcrDyxHL.8t.5llp","ns":""},"count":1},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:51.123 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-23 2020-12-05T19:15:51.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:51.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:51.125 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:51.165 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:51.165 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:51.388 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:51.389 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-23 2020-12-05T19:15:51.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:51.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:51.390 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:51.477 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:51.477 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:51.720 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"U5IsFzxpWezYRYGUcrDyxHL.8t.5llp","ns":""},"count":1},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:51.721 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-23 2020-12-05T19:15:51.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:51.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:51.722 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:51.803 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:51.804 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:52.319 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:52.320 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-23 2020-12-05T19:15:52.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:52.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:52.323 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:52.368 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:52.368 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:52.646 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:52.647 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-23 2020-12-05T19:15:52.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:52.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:52.649 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:52.731 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:52.731 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:52.984 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:52.984 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-23 2020-12-05T19:15:52.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:52.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:52.986 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:53.030 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:53.031 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:53.288 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"U5IsFzxpWezYRYGUcrDyxHL.8t.5llp","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:53.288 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-23 2020-12-05T19:15:53.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:53.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:53.290 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:53.571 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:53.572 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-23 2020-12-05T19:15:53.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:53.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:53.575 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:53.619 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:53.619 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:53.875 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"U5IsFzxpWezYRYGUcrDyxHL.8t.5llp","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:53.875 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-23 2020-12-05T19:15:53.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:53.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:53.877 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:53.963 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:53.963 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:54.189 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.16.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:54.189 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-23 2020-12-05T19:15:54.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:54.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:54.191 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:54.272 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:54.273 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:54.525 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"U5IsFzxpWezYRYGUcrDyxHL.8t.5llp","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:54.525 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-23 2020-12-05T19:15:54.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:54.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:54.527 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:54.607 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:54.608 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:54.843 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.16.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:15:54.844 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-23 2020-12-05T19:15:54.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:54.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:54.845 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:55.159 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"U5IsFzxpWezYRYGUcrDyxHL.8t.5llp","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:15:55.160 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-23 2020-12-05T19:15:55.160 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-23 zones={test-zone2, test-zone1} 2020-12-05T19:15:56.991 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:57.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:57.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:57.001 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:57.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:57.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:57.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:57.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:57.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:57.202 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-23 zones={test-zone2, test-zone1} 2020-12-05T19:15:57.202 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-23 zones={test-zone3, test-zone1} 2020-12-05T19:15:57.761 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:57.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:57.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:57.815 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:57.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:57.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:57.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:57.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:57.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:57.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-23 zones={test-zone3, test-zone1} 2020-12-05T19:15:57.969 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-23 zones={test-zone3, test-zone2} 2020-12-05T19:15:57.980 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:15:58.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:58.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:58.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:15:58.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:58.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:15:58.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:58.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:58.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:15:58.184 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-23 zones={test-zone3, test-zone2} 2020-12-05T19:15:58.185 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-12-05T19:15:58.186 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-24 2020-12-05T19:15:58.218 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-25 2020-12-05T19:15:58.273 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-26 2020-12-05T19:15:58.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:58.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:58.419 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:58.456 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:58.456 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:58.660 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:15:58.697 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195758.409925_259.1","last_update":"2020-12-05T19:15:58.409925Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:15:58.698 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:15:58.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:58.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:15:58.700 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:15:58.740 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:15:58.741 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:15:58.977 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195727.946934_235.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:27.946934Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:15:58.978 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195758.409925_259.1 target=1_1607195727.946934_235.1 2020-12-05T19:16:03.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:03.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:03.981 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:04.022 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:04.022 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:04.268 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195758.409925_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:58.409925Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:04.269 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:16:04.270 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:16:04.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:04.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:04.272 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:04.327 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:04.327 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:04.559 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195758.369799_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:58.369799Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:04.560 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1607195758.409925_259.1 target=1_1607195758.369799_255.1 2020-12-05T19:16:09.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:09.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:09.562 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:09.619 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:09.619 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:09.852 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195758.409925_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:58.409925Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:09.853 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:16:09.854 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:16:09.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:09.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:09.855 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:09.902 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:09.903 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:10.176 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195758.409925_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:15:58.409925Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:10.176 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:16:11.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:11.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:11.716 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:11.755 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:11.755 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:11.993 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.51.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:11.994 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-24 2020-12-05T19:16:11.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:11.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:11.995 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:12.039 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:12.040 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:12.287 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.51.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:12.288 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-24 2020-12-05T19:16:12.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:12.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:12.290 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:12.331 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:12.332 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:12.556 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.51.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:12.556 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-24 2020-12-05T19:16:12.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:12.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:12.558 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:12.638 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:12.639 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:12.861 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.51.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:12.861 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-24 2020-12-05T19:16:12.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:12.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:12.864 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:12.940 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:12.941 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:13.171 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.51.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:13.171 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-24 2020-12-05T19:16:13.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:13.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:13.173 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:13.218 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:13.218 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:13.494 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.51.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:13.494 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-24 2020-12-05T19:16:13.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:13.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:13.496 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:13.539 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:13.540 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:13.759 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:13.759 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-24 2020-12-05T19:16:13.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:13.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:13.761 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:13.810 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:13.810 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:14.031 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:14.031 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-24 2020-12-05T19:16:14.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:14.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:14.033 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:14.074 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:14.075 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:14.316 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:14.316 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-24 2020-12-05T19:16:14.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:14.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:14.319 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:14.363 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:14.363 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:14.606 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:14.607 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-24 2020-12-05T19:16:14.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:14.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:14.608 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:14.694 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:14.695 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:14.922 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:14.923 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-24 2020-12-05T19:16:14.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:14.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:14.925 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:15.008 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:15.009 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:15.271 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:15.272 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-24 2020-12-05T19:16:15.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:15.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:15.274 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:15.354 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:15.355 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:15.587 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:15.588 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-24 2020-12-05T19:16:15.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:15.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:15.589 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:15.634 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:15.634 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:15.893 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:15.893 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-24 2020-12-05T19:16:15.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-24 zones={test-zone2, test-zone1} 2020-12-05T19:16:17.318 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:16:17.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:17.326 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:16:17.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:17.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:16:17.445 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-24 zones={test-zone2, test-zone1} 2020-12-05T19:16:17.446 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-24 zones={test-zone3, test-zone1} 2020-12-05T19:16:17.771 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:16:17.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:17.823 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:16:17.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:17.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:16:17.900 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-24 zones={test-zone3, test-zone1} 2020-12-05T19:16:17.901 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-24 zones={test-zone3, test-zone2} 2020-12-05T19:16:17.910 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:16:17.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:17.963 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:16:17.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:18.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:16:18.264 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-24 zones={test-zone3, test-zone2} 2020-12-05T19:16:18.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:18.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:18.265 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:18.305 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:18.306 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:18.546 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:18.547 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-25 2020-12-05T19:16:18.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:18.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:18.549 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:18.593 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:18.593 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:18.847 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:18.847 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-25 2020-12-05T19:16:18.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:18.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:18.849 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:18.891 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:18.891 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:19.132 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:19.133 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-25 2020-12-05T19:16:19.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:19.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:19.135 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:19.217 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:19.218 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:19.464 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:19.465 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-25 2020-12-05T19:16:19.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:19.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:19.466 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:19.546 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:19.547 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:19.789 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:19.789 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-25 2020-12-05T19:16:19.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:19.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:19.791 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:20.106 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.28.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:20.107 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-25 2020-12-05T19:16:20.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:20.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:20.108 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:20.194 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:20.194 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:20.444 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.43.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:20.444 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-25 2020-12-05T19:16:20.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:20.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:20.446 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:20.490 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:20.490 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:20.752 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.43.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:20.753 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-25 2020-12-05T19:16:20.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:20.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:20.755 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:21.039 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.43.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:21.040 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-25 2020-12-05T19:16:21.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:21.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:21.042 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:21.086 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:21.086 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:21.339 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.43.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:21.340 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-25 2020-12-05T19:16:21.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:21.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:21.341 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:21.420 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:21.421 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:21.658 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.41.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:21.658 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-25 2020-12-05T19:16:21.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:21.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:21.660 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:21.744 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:21.744 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:21.986 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:21.986 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-25 2020-12-05T19:16:21.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:21.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:21.989 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:22.071 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:22.071 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:22.290 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.41.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:22.291 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-25 2020-12-05T19:16:22.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:22.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:22.292 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:22.592 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:22.593 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-25 2020-12-05T19:16:22.593 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-25 zones={test-zone2, test-zone1} 2020-12-05T19:16:23.581 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:16:23.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:23.587 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:16:23.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:23.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:16:23.704 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-25 zones={test-zone2, test-zone1} 2020-12-05T19:16:23.705 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-25 zones={test-zone3, test-zone1} 2020-12-05T19:16:23.858 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:16:23.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:23.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:16:23.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:23.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:16:23.986 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-25 zones={test-zone3, test-zone1} 2020-12-05T19:16:23.987 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-25 zones={test-zone3, test-zone2} 2020-12-05T19:16:23.998 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:16:24.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:24.048 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:16:24.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:24.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:16:24.123 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-25 zones={test-zone3, test-zone2} 2020-12-05T19:16:24.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:24.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:24.124 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:24.164 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:24.165 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:24.402 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.30.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:24.402 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-26 2020-12-05T19:16:24.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:24.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:24.404 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:24.700 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:24.700 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-26 2020-12-05T19:16:24.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:24.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:24.702 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:24.743 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:24.744 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:24.987 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.30.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:24.987 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-26 2020-12-05T19:16:24.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:24.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:24.989 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:25.071 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:25.071 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:25.336 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:25.337 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-26 2020-12-05T19:16:25.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:25.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:25.338 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:25.419 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:25.419 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:25.659 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.30.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:25.660 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-26 2020-12-05T19:16:25.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:25.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:25.661 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:25.705 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:25.971 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.30.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:25.972 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-26 2020-12-05T19:16:25.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:25.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:25.973 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:26.053 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:26.054 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:26.299 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.45.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:26.300 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-26 2020-12-05T19:16:26.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:26.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:26.302 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:26.346 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:26.347 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:26.607 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:26.607 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-26 2020-12-05T19:16:26.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:26.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:26.609 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:26.650 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:26.651 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:26.886 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.45.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:26.887 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-26 2020-12-05T19:16:26.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:26.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:26.889 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:26.930 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:26.931 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:27.204 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:27.204 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-26 2020-12-05T19:16:27.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:27.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:27.206 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:27.286 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:27.286 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:27.531 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.39.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:27.532 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-26 2020-12-05T19:16:27.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:27.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:27.534 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:27.621 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:27.622 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:27.878 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.39.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:27.878 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-26 2020-12-05T19:16:27.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:27.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:27.880 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:27.962 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:27.963 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:28.200 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.39.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:16:28.201 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-26 2020-12-05T19:16:28.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:28.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:28.202 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:28.246 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:28.521 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.39.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:16:28.522 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-26 2020-12-05T19:16:28.522 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-26 zones={test-zone2, test-zone1} 2020-12-05T19:16:29.725 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:16:29.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:29.734 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:16:29.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:29.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:16:29.850 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-26 zones={test-zone2, test-zone1} 2020-12-05T19:16:29.851 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-26 zones={test-zone3, test-zone1} 2020-12-05T19:16:30.807 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:16:30.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:30.859 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:16:30.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:30.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:16:30.935 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-26 zones={test-zone3, test-zone1} 2020-12-05T19:16:30.935 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-26 zones={test-zone3, test-zone2} 2020-12-05T19:16:31.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:16:31.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:31.423 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:16:31.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-12-05T19:16:31.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-12-05T19:16:31.540 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-26 zones={test-zone3, test-zone2} 2020-12-05T19:16:31.541 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-12-05T19:16:31.541 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-27 2020-12-05T19:16:31.575 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-28 2020-12-05T19:16:31.641 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-29 2020-12-05T19:16:31.894 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-12-05T19:16:31.895 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-30 2020-12-05T19:16:31.929 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-31 2020-12-05T19:16:31.995 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-32 2020-12-05T19:16:32.242 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-12-05T19:16:32.243 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-33 2020-12-05T19:16:32.279 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-34 2020-12-05T19:16:32.347 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-35 2020-12-05T19:16:34.942 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-12-05T19:16:34.943 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-36 2020-12-05T19:16:35.761 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-37 2020-12-05T19:16:36.437 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-38 2020-12-05T19:16:37.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:37.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:37.209 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:37.242 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:37.242 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:37.437 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:16:37.472 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195797.184194_325.1","last_update":"2020-12-05T19:16:37.184194Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:16:37.472 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:16:37.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:37.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:37.474 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:37.515 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:37.516 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:37.731 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195796.408510_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:36.408510Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:37.731 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195797.184194_325.1 target=1_1607195796.408510_321.1 2020-12-05T19:16:42.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:42.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:42.734 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:43.014 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195797.184194_325.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:37.184194Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:43.014 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:16:43.015 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:16:43.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:43.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:43.016 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:43.058 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:43.058 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:43.296 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195797.184194_325.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:37.184194Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:43.297 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:16:43.297 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:16:43.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:43.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:43.299 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:43.339 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:43.340 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:43.617 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195797.184194_325.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:37.184194Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:43.617 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:16:44.944 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-12-05T19:16:44.944 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-39 2020-12-05T19:16:44.973 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-40 2020-12-05T19:16:45.039 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-41 2020-12-05T19:16:45.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:45.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:45.106 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:45.140 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:45.140 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:45.335 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=cb156068-23b3-4eca-8803-e19dbeb79445 2020-12-05T19:16:45.370 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195426.378999_125.1","last_update":"2020-12-05T19:10:26.378999Z"},{"marker":"1_1607195805.076974_337.1","last_update":"2020-12-05T19:16:45.076974Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:16:45.371 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:16:45.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:45.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:45.372 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:45.414 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:45.660 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195805.068677_335.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:45.068677Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:45.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607195805.076974_337.1 target=1_1607195805.068677_335.1 2020-12-05T19:16:50.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:50.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:50.664 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:50.706 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:50.706 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:50.958 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195805.076974_337.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:45.076974Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:50.958 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:16:50.959 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:16:50.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:50.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:50.960 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:50.999 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:51.000 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:51.235 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195805.076974_337.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:45.076974Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:51.236 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:16:51.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:16:51.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:51.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:16:51.239 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:16:51.278 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:51.279 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:51.559 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"cb156068-23b3-4eca-8803-e19dbeb79445","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195805.076974_337.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:45.076974Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:16:51.560 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:16:51.560 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-12-05T19:16:51.561 INFO:teuthology.orchestra.run:waiting for 300 2020-12-05T19:16:51.561 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:16:51.553+0000 7f349f7fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 14838) UID: 0 2020-12-05T19:16:51.562 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:16:51.553+0000 7f34bb3a90c0 -1 shutting down 2020-12-05T19:16:53.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-12-05T19:16:57.662 INFO:tasks.rgw.c1.client.1:Stopped 2020-12-05T19:16:57.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:16:57.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:16:57.664 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:16:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:57.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:16:57.730+0000 7f2d5abcb080 0 NOTICE: overriding master zone: e622b2e3-1a91-4dc1-b31e-43ede659e70f 2020-12-05T19:16:57.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:16:57.730+0000 7f2d5abcb080 0 failed reading obj info from .rgw.root:zone_info.e622b2e3-1a91-4dc1-b31e-43ede659e70f: (2) No such file or directory 2020-12-05T19:16:57.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:16:57.734+0000 7f2d5abcb080 0 WARNING: could not read zone params for zone id=e622b2e3-1a91-4dc1-b31e-43ede659e70f name=test-zone1 2020-12-05T19:16:57.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:16:57.734+0000 7f2d5abcb080 0 failed reading obj info from .rgw.root:zone_info.ecced906-ad37-41c3-ad42-864c1e77463f: (2) No such file or directory 2020-12-05T19:16:57.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:16:57.734+0000 7f2d5abcb080 0 WARNING: could not read zone params for zone id=ecced906-ad37-41c3-ad42-864c1e77463f name=test-zone3 2020-12-05T19:16:57.746 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:16:57.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:16:57.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:16:57.748 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:16:57.825 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:16:57.825 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:16:58.144 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"88dc4e82-8825-4cad-908f-99aa84fb6061","epoch":1,"predecessor_uuid":"cb156068-23b3-4eca-8803-e19dbeb79445","sync_status":["1_1607195426.378999_125.1","1_1607195805.076974_337.1","",""],"period_map":{"id":"88dc4e82-8825-4cad-908f-99aa84fb6061","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"3ee88d97-6118-4d9f-87f0-34d91092c64f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"3ee88d97-6118-4d9f-87f0-34d91092c64f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":3} 2020-12-05T19:16:58.144 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-12-05T19:16:59.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-12-05T19:17:58.146 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-42 2020-12-05T19:17:58.241 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-43 2020-12-05T19:17:58.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:17:58.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:17:58.281 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:17:58.317 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:17:58.318 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:17:58.535 INFO:teuthology.orchestra.run.smithi023.stderr:No --period given, using current period=88dc4e82-8825-4cad-908f-99aa84fb6061 2020-12-05T19:17:58.572 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607195878.275808_415.1","last_update":"2020-12-05T19:17:58.275808Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:17:58.572 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-12-05T19:17:58.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:17:58.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:17:58.574 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:17:58.615 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:17:58.615 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:17:58.871 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"88dc4e82-8825-4cad-908f-99aa84fb6061","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2020-12-05T19:17:58.871 WARNING:rgw_multi.tests:zone test-zone1 behind master: shard=1 master=1_1607195878.275808_415.1 target= 2020-12-05T19:18:03.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:18:03.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:18:03.874 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:18:03.916 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:18:03.917 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:18:04.167 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"88dc4e82-8825-4cad-908f-99aa84fb6061","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"1_1607195878.275808_415.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:17:58.275808Z","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2020-12-05T19:18:04.168 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-12-05T19:18:04.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:18:04.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:18:04.170 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:18:04.247 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:18:04.247 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:18:04.274 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:18:04.267+0000 7fed126fe080 0 NOTICE: overriding master zone: 3ee88d97-6118-4d9f-87f0-34d91092c64f 2020-12-05T19:18:04.275 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:18:04.271+0000 7fed126fe080 0 failed reading obj info from .rgw.root:zone_info.3ee88d97-6118-4d9f-87f0-34d91092c64f: (2) No such file or directory 2020-12-05T19:18:04.275 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:18:04.271+0000 7fed126fe080 0 WARNING: could not read zone params for zone id=3ee88d97-6118-4d9f-87f0-34d91092c64f name=test-zone2 2020-12-05T19:18:04.276 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:18:04.271+0000 7fed126fe080 0 failed reading obj info from .rgw.root:zone_info.90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d: (2) No such file or directory 2020-12-05T19:18:04.276 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:18:04.271+0000 7fed126fe080 0 WARNING: could not read zone params for zone id=90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d name=test-zone4 2020-12-05T19:18:04.291 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:18:04.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:18:04.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:18:04.293 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:18:04.372 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:18:04.372 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:18:04.733 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","epoch":1,"predecessor_uuid":"88dc4e82-8825-4cad-908f-99aa84fb6061","sync_status":["","1_1607195878.275808_415.1","",""],"period_map":{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":4} 2020-12-05T19:18:04.733 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-12-05T19:19:04.734 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-44 2020-12-05T19:19:04.782 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-45 2020-12-05T19:19:04.853 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-12-05T19:19:04.854 INFO:teuthology.orchestra.run.smithi185:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-12-05T19:19:04.857 INFO:tasks.rgw.c1.client.1:Started 2020-12-05T19:19:04.857 INFO:teuthology.orchestra.run.smithi185:> curl http://smithi185.front.sepia.ceph.com:8001 2020-12-05T19:19:04.870 INFO:teuthology.orchestra.run.smithi185.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:19:04.871 INFO:teuthology.orchestra.run.smithi185.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:19:04.875 INFO:teuthology.orchestra.run.smithi185.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi185.front.sepia.ceph.com port 8001: Connection refused 2020-12-05T19:19:04.875 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-12-05T19:19:05.876 INFO:teuthology.orchestra.run.smithi185:> curl http://smithi185.front.sepia.ceph.com:8001 2020-12-05T19:19:05.893 INFO:teuthology.orchestra.run.smithi185.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-12-05T19:19:05.893 INFO:teuthology.orchestra.run.smithi185.stderr: Dload Upload Total Spent Left Speed 2020-12-05T19:19:05.897 INFO:teuthology.orchestra.run.smithi185.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 53500 2020-12-05T19:19:05.898 INFO:teuthology.orchestra.run.smithi185.stdout:anonymous 2020-12-05T19:19:05.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:05.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:05.900 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:05.978 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:05.978 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:06.187 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:19:06.219 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607195944.825629_254.1","last_update":"2020-12-05T19:19:04.825629Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:19:06.220 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:19:06.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:06.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:06.221 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:06.262 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:06.263 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:06.501 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195944.825629_254.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:04.825629Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:19:06.501 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:19:06.502 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:19:06.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:06.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:06.503 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:06.543 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:06.544 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:06.795 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195944.825629_254.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:04.825629Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:19:06.796 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:19:06.797 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:19:06.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:06.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:06.798 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:06.838 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:06.838 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:07.118 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195944.825629_254.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:04.825629Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:19:07.119 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:19:07.119 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-39 zones={test-zone2, test-zone1} 2020-12-05T19:19:08.234 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:08.242 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:08.346 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-39 zones={test-zone2, test-zone1} 2020-12-05T19:19:08.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-39 zones={test-zone3, test-zone1} 2020-12-05T19:19:08.368 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:08.422 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:08.490 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-39 zones={test-zone3, test-zone1} 2020-12-05T19:19:08.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-39 zones={test-zone1, test-zone2} 2020-12-05T19:19:08.502 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:08.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:08.620 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-39 zones={test-zone1, test-zone2} 2020-12-05T19:19:08.621 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-39 zones={test-zone3, test-zone2} 2020-12-05T19:19:08.631 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:08.682 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:08.749 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-39 zones={test-zone3, test-zone2} 2020-12-05T19:19:08.750 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-39 zones={test-zone1, test-zone3} 2020-12-05T19:19:08.764 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:08.810 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:08.877 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-39 zones={test-zone1, test-zone3} 2020-12-05T19:19:08.878 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-39 zones={test-zone2, test-zone3} 2020-12-05T19:19:08.889 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:08.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:09.010 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-39 zones={test-zone2, test-zone3} 2020-12-05T19:19:09.011 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-40 zones={test-zone2, test-zone1} 2020-12-05T19:19:09.024 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:09.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:09.138 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-40 zones={test-zone2, test-zone1} 2020-12-05T19:19:09.139 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-40 zones={test-zone3, test-zone1} 2020-12-05T19:19:09.150 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:09.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:09.270 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-40 zones={test-zone3, test-zone1} 2020-12-05T19:19:09.271 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-40 zones={test-zone1, test-zone2} 2020-12-05T19:19:09.283 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:09.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:09.397 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-40 zones={test-zone1, test-zone2} 2020-12-05T19:19:09.398 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-40 zones={test-zone3, test-zone2} 2020-12-05T19:19:09.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:09.458 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:09.520 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-40 zones={test-zone3, test-zone2} 2020-12-05T19:19:09.520 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-40 zones={test-zone1, test-zone3} 2020-12-05T19:19:09.533 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:09.582 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:09.645 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-40 zones={test-zone1, test-zone3} 2020-12-05T19:19:09.646 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-40 zones={test-zone2, test-zone3} 2020-12-05T19:19:09.657 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:09.706 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:09.778 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-40 zones={test-zone2, test-zone3} 2020-12-05T19:19:09.779 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-41 zones={test-zone2, test-zone1} 2020-12-05T19:19:09.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:09.842 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:09.911 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-41 zones={test-zone2, test-zone1} 2020-12-05T19:19:09.912 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-41 zones={test-zone3, test-zone1} 2020-12-05T19:19:09.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:09.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:10.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-41 zones={test-zone3, test-zone1} 2020-12-05T19:19:10.038 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-41 zones={test-zone1, test-zone2} 2020-12-05T19:19:10.049 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:10.098 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:10.165 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-41 zones={test-zone1, test-zone2} 2020-12-05T19:19:10.165 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-41 zones={test-zone3, test-zone2} 2020-12-05T19:19:10.176 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:10.226 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:10.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-41 zones={test-zone3, test-zone2} 2020-12-05T19:19:10.293 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-41 zones={test-zone1, test-zone3} 2020-12-05T19:19:10.307 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:10.358 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:10.423 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-41 zones={test-zone1, test-zone3} 2020-12-05T19:19:10.424 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-41 zones={test-zone2, test-zone3} 2020-12-05T19:19:10.435 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:10.486 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:10.556 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-41 zones={test-zone2, test-zone3} 2020-12-05T19:19:10.557 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-42 zones={test-zone2, test-zone1} 2020-12-05T19:19:10.567 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:10.618 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:10.681 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-42 zones={test-zone2, test-zone1} 2020-12-05T19:19:10.682 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-42 zones={test-zone3, test-zone1} 2020-12-05T19:19:10.693 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:10.742 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:10.811 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-42 zones={test-zone3, test-zone1} 2020-12-05T19:19:10.811 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-42 zones={test-zone1, test-zone2} 2020-12-05T19:19:10.822 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:10.874 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:10.945 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-42 zones={test-zone1, test-zone2} 2020-12-05T19:19:10.945 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-42 zones={test-zone3, test-zone2} 2020-12-05T19:19:10.954 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:11.002 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:11.065 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-42 zones={test-zone3, test-zone2} 2020-12-05T19:19:11.066 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-42 zones={test-zone1, test-zone3} 2020-12-05T19:19:11.079 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:11.130 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:11.194 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-42 zones={test-zone1, test-zone3} 2020-12-05T19:19:11.195 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-42 zones={test-zone2, test-zone3} 2020-12-05T19:19:11.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:11.250 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:11.318 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-42 zones={test-zone2, test-zone3} 2020-12-05T19:19:11.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-43 zones={test-zone2, test-zone1} 2020-12-05T19:19:11.330 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:11.382 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:11.451 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-43 zones={test-zone2, test-zone1} 2020-12-05T19:19:11.452 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-43 zones={test-zone3, test-zone1} 2020-12-05T19:19:11.461 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:11.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:11.579 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-43 zones={test-zone3, test-zone1} 2020-12-05T19:19:11.580 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-43 zones={test-zone1, test-zone2} 2020-12-05T19:19:11.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:11.638 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:11.705 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-43 zones={test-zone1, test-zone2} 2020-12-05T19:19:11.705 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-43 zones={test-zone3, test-zone2} 2020-12-05T19:19:11.717 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:11.766 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:11.829 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-43 zones={test-zone3, test-zone2} 2020-12-05T19:19:11.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-43 zones={test-zone1, test-zone3} 2020-12-05T19:19:11.842 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:11.890 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:11.953 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-43 zones={test-zone1, test-zone3} 2020-12-05T19:19:11.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-43 zones={test-zone2, test-zone3} 2020-12-05T19:19:11.964 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:12.014 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:12.083 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-43 zones={test-zone2, test-zone3} 2020-12-05T19:19:12.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-44 zones={test-zone2, test-zone1} 2020-12-05T19:19:12.095 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:12.142 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:12.211 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-44 zones={test-zone2, test-zone1} 2020-12-05T19:19:12.211 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-44 zones={test-zone3, test-zone1} 2020-12-05T19:19:12.221 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:12.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:12.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-44 zones={test-zone3, test-zone1} 2020-12-05T19:19:12.338 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-44 zones={test-zone1, test-zone2} 2020-12-05T19:19:12.348 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:12.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:12.465 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-44 zones={test-zone1, test-zone2} 2020-12-05T19:19:12.465 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-44 zones={test-zone3, test-zone2} 2020-12-05T19:19:12.477 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:12.530 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:12.597 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-44 zones={test-zone3, test-zone2} 2020-12-05T19:19:12.598 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-44 zones={test-zone1, test-zone3} 2020-12-05T19:19:12.610 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:12.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:12.725 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-44 zones={test-zone1, test-zone3} 2020-12-05T19:19:12.726 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-44 zones={test-zone2, test-zone3} 2020-12-05T19:19:12.737 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:12.786 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:12.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-44 zones={test-zone2, test-zone3} 2020-12-05T19:19:12.854 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-45 zones={test-zone2, test-zone1} 2020-12-05T19:19:12.864 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:12.914 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:12.985 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-45 zones={test-zone2, test-zone1} 2020-12-05T19:19:12.985 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-45 zones={test-zone3, test-zone1} 2020-12-05T19:19:12.994 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:13.046 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:13.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-45 zones={test-zone3, test-zone1} 2020-12-05T19:19:13.110 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-45 zones={test-zone1, test-zone2} 2020-12-05T19:19:13.121 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:13.166 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:13.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-45 zones={test-zone1, test-zone2} 2020-12-05T19:19:13.230 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-45 zones={test-zone3, test-zone2} 2020-12-05T19:19:13.240 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:13.286 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:13.349 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-45 zones={test-zone3, test-zone2} 2020-12-05T19:19:13.350 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-45 zones={test-zone1, test-zone3} 2020-12-05T19:19:13.362 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:13.410 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:13.473 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-45 zones={test-zone1, test-zone3} 2020-12-05T19:19:13.473 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-45 zones={test-zone2, test-zone3} 2020-12-05T19:19:13.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:19:13.530 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:19:13.595 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-45 zones={test-zone2, test-zone3} 2020-12-05T19:19:13.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:13.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:13.596 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period cb156068-23b3-4eca-8803-e19dbeb79445 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:13.634 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:13.634 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:13.925 INFO:teuthology.orchestra.run.smithi185.stdout:[{"id":"1_1607195257.607529_44.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-12-05T19:07:37.607529336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QzWxyJmSnbzgxXwE-SYm0Au","ver":1},"status":{"status":"write"}}},{"id":"1_1607195257.616573_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-12-05T19:07:37.616573496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QzWxyJmSnbzgxXwE-SYm0Au","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.004271_46.1","section":"bucket.instance","name":"tazcyr-1:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1","timestamp":"2020-12-05T19:09:40.004271326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzba4sBo5g6GI2dMc2uNhJS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.016937_47.1","section":"bucket.instance","name":"tazcyr-1:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1","timestamp":"2020-12-05T19:09:40.016937469Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzba4sBo5g6GI2dMc2uNhJS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.021417_48.1","section":"bucket","name":"tazcyr-1","timestamp":"2020-12-05T19:09:40.021417606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VyXMtA4g5xk2Opu-uU28KKS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.024544_49.1","section":"bucket","name":"tazcyr-1","timestamp":"2020-12-05T19:09:40.024544422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VyXMtA4g5xk2Opu-uU28KKS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.046042_50.1","section":"bucket.instance","name":"tazcyr-2:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2","timestamp":"2020-12-05T19:09:40.046042905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KP2FkRxiLEAiJe0MTCH9I4b","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.051202_51.1","section":"bucket.instance","name":"tazcyr-2:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2","timestamp":"2020-12-05T19:09:40.051202070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KP2FkRxiLEAiJe0MTCH9I4b","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.052778_52.1","section":"bucket","name":"tazcyr-2","timestamp":"2020-12-05T19:09:40.052778306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WUyHO6QQbtRgQbl6LIL5fmT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.057318_53.1","section":"bucket","name":"tazcyr-2","timestamp":"2020-12-05T19:09:40.057318427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WUyHO6QQbtRgQbl6LIL5fmT","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195383.710306_54.1","section":"bucket.instance","name":"tazcyr-3:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3","timestamp":"2020-12-05T19:09:43.710306623Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mYlUpB34g7c36jCJJY7xASF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195383.720265_55.1","section":"bucket.instance","name":"tazcyr-3:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3","timestamp":"2020-12-05T19:09:43.720265984Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mYlUpB34g7c36jCJJY7xASF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195383.722055_56.1","section":"bucket","name":"tazcyr-3","timestamp":"2020-12-05T19:09:43.722055553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YJJrKeypq2XGKHx5hyNMCIx","ver":1},"status":{"status":"write"}}},{"id":"1_1607195383.731707_57.1","section":"bucket","name":"tazcyr-3","timestamp":"2020-12-05T19:09:43.731707462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YJJrKeypq2XGKHx5hyNMCIx","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.335546_60.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4","timestamp":"2020-12-05T19:09:51.335546801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L_jTexV2QOKdHJrFnfn7kbh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.342055_61.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4","timestamp":"2020-12-05T19:09:51.342055564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L_jTexV2QOKdHJrFnfn7kbh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.344193_62.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:09:51.344193062Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jLnWUWkBmUYtUiCVFVQfp-b","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.351287_63.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:09:51.351287484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jLnWUWkBmUYtUiCVFVQfp-b","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.373162_64.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5","timestamp":"2020-12-05T19:09:51.373162509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E8T9igSLKPGjPwqoSjLEp_k","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.378587_65.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5","timestamp":"2020-12-05T19:09:51.378587248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E8T9igSLKPGjPwqoSjLEp_k","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.381243_66.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:09:51.381243284Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rz7EK0Cpc8qessr8O7Tn5iH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.388606_67.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:09:51.388606811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rz7EK0Cpc8qessr8O7Tn5iH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.437812_68.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6","timestamp":"2020-12-05T19:09:51.437812836Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZX-6V2b9MxDZwTcrSqix7n6","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.441771_69.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6","timestamp":"2020-12-05T19:09:51.441771244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZX-6V2b9MxDZwTcrSqix7n6","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.444005_70.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:09:51.444005575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6jyy8rxZVWt7mkAC_xVvmdF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.448654_71.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:09:51.448654007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6jyy8rxZVWt7mkAC_xVvmdF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.085465_72.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.7","timestamp":"2020-12-05T19:10:04.085465407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Cd-JV10QVaCV4xzrR7e8Cb","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.089157_73.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.7","timestamp":"2020-12-05T19:10:04.089157232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Cd-JV10QVaCV4xzrR7e8Cb","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.091040_74.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.091040641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ibQYDdoT09BZvl4zzz8MPh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.094388_75.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.094388936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ibQYDdoT09BZvl4zzz8MPh","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.124511_76.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.8","timestamp":"2020-12-05T19:10:04.124511324Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9u5vcAmfWRm3ybgEgHDxyJ7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.129450_77.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.8","timestamp":"2020-12-05T19:10:04.129450232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9u5vcAmfWRm3ybgEgHDxyJ7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.132112_78.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.132112638Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QhkUsfUtoco38o0vRSTiqJm","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.137475_79.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.137475791Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QhkUsfUtoco38o0vRSTiqJm","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.164520_80.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.9","timestamp":"2020-12-05T19:10:04.164520302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8G8bhkxk9kDUQA-JQCo8M0","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.169210_81.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.9","timestamp":"2020-12-05T19:10:04.169210675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8G8bhkxk9kDUQA-JQCo8M0","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.171105_82.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.171105195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ERy_2HXt0qrGTMck-HhMHoA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.174573_83.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.174573484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ERy_2HXt0qrGTMck-HhMHoA","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.202373_84.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.10","timestamp":"2020-12-05T19:10:04.202373262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjYrzFHFxksxKI8RLAch6d7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.206658_85.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.10","timestamp":"2020-12-05T19:10:04.206658084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjYrzFHFxksxKI8RLAch6d7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.209421_86.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.209421800Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_peuGzSEo8-wqUKsHfkgtZvP","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.212892_87.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.212892151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_peuGzSEo8-wqUKsHfkgtZvP","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.271570_88.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.11","timestamp":"2020-12-05T19:10:04.271570054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GIGteFOTnsUR-BSdJ7KuGyi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.276571_89.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.11","timestamp":"2020-12-05T19:10:04.276571979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GIGteFOTnsUR-BSdJ7KuGyi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.278943_90.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.278943293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZCB1N_YTX9IpoOGN96l67YT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.286241_91.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.286241492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZCB1N_YTX9IpoOGN96l67YT","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.393793_92.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.12","timestamp":"2020-12-05T19:10:04.393793199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wZ6ND_m9qn3va88zzyFK_lh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.398122_93.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.12","timestamp":"2020-12-05T19:10:04.398122874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wZ6ND_m9qn3va88zzyFK_lh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.400992_94.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.400992194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bvtob7IKitpBTl1g-h1fDWw","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.407785_95.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.407785961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bvtob7IKitpBTl1g-h1fDWw","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.516948_96.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.13","timestamp":"2020-12-05T19:10:04.516948552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_S9v1ugs3p0bAN19LWmAKo","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.521889_97.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.13","timestamp":"2020-12-05T19:10:04.521889148Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_S9v1ugs3p0bAN19LWmAKo","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.524060_98.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.524060551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRn-G8myGtlhg91Dk-Xg1zs","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.528641_99.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.528641393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRn-G8myGtlhg91Dk-Xg1zs","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.572841_100.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.14","timestamp":"2020-12-05T19:10:04.572841515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VKyqbVNoVDGlDXXzerwmsua","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.577361_101.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.14","timestamp":"2020-12-05T19:10:04.577361576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VKyqbVNoVDGlDXXzerwmsua","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.579161_102.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.579161032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_egt4Jp_vRt2MTLqMPIEdrsz","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.582561_103.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.582561935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_egt4Jp_vRt2MTLqMPIEdrsz","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.682129_104.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.15","timestamp":"2020-12-05T19:10:04.682129773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TiQoTfba0kD_Nn8vVWkQDe8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.689318_105.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.15","timestamp":"2020-12-05T19:10:04.689318302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TiQoTfba0kD_Nn8vVWkQDe8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.691755_106.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.691755301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJgnpoAQLxQuG357X5jW8Wb","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.697154_107.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.697154104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJgnpoAQLxQuG357X5jW8Wb","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195417.967417_108.1","section":"bucket.instance","name":"tazcyr-7:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16","timestamp":"2020-12-05T19:10:17.967417000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PK9gOINKQY1odbbT4wek8C_","ver":1},"status":{"status":"write"}}},{"id":"1_1607195417.971501_109.1","section":"bucket.instance","name":"tazcyr-7:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16","timestamp":"2020-12-05T19:10:17.971501874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PK9gOINKQY1odbbT4wek8C_","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195417.973539_110.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:17.973539330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195417.977586_111.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:17.977586121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195417.998346_112.1","section":"bucket.instance","name":"tazcyr-8:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17","timestamp":"2020-12-05T19:10:17.998346857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i7BUcpgTpgUsW-HrIAVrF6r","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.003566_113.1","section":"bucket.instance","name":"tazcyr-8:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17","timestamp":"2020-12-05T19:10:18.003566781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i7BUcpgTpgUsW-HrIAVrF6r","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.005584_114.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:18.005584098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.009519_115.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:18.009519950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.057689_116.1","section":"bucket.instance","name":"tazcyr-9:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18","timestamp":"2020-12-05T19:10:18.057689828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BCybjCVg7kHS2ZQlWPWBlz","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.061844_117.1","section":"bucket.instance","name":"tazcyr-9:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18","timestamp":"2020-12-05T19:10:18.061844526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BCybjCVg7kHS2ZQlWPWBlz","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.063969_118.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:18.063969600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.067947_119.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:18.067947008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195426.216787_120.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:26.216787778Z","data":{"read_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.221990_121.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:26.221990508Z","data":{"read_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195426.277724_122.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:26.277724745Z","data":{"read_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.284587_123.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:26.284587333Z","data":{"read_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195426.373298_124.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:26.373298757Z","data":{"read_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.378999_125.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:26.378999520Z","data":{"read_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195433.462730_126.1","section":"bucket.instance","name":"tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19","timestamp":"2020-12-05T19:10:33.462730080Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_52wD5vB-WvWwe7ywIABs_eP","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.467706_127.1","section":"bucket.instance","name":"tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19","timestamp":"2020-12-05T19:10:33.467706513Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_52wD5vB-WvWwe7ywIABs_eP","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.470216_128.1","section":"bucket","name":"tazcyr-10","timestamp":"2020-12-05T19:10:33.470216766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gF0E0avjnogGfUjbNKC363v","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.476060_129.1","section":"bucket","name":"tazcyr-10","timestamp":"2020-12-05T19:10:33.476060218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gF0E0avjnogGfUjbNKC363v","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.501303_130.1","section":"bucket.instance","name":"tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20","timestamp":"2020-12-05T19:10:33.501303520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IpmsnwQB3g2CXO9fV5tmzxe","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.505700_131.1","section":"bucket.instance","name":"tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20","timestamp":"2020-12-05T19:10:33.505700636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IpmsnwQB3g2CXO9fV5tmzxe","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.510386_132.1","section":"bucket","name":"tazcyr-11","timestamp":"2020-12-05T19:10:33.510386421Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCLh27ru7alDAFNCjpuYcbo","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.515084_133.1","section":"bucket","name":"tazcyr-11","timestamp":"2020-12-05T19:10:33.515084294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCLh27ru7alDAFNCjpuYcbo","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.563793_134.1","section":"bucket.instance","name":"tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21","timestamp":"2020-12-05T19:10:33.563793962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oEVFbZgDKIx5z8F7aheS42B","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.568424_135.1","section":"bucket.instance","name":"tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21","timestamp":"2020-12-05T19:10:33.568424303Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oEVFbZgDKIx5z8F7aheS42B","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.570642_136.1","section":"bucket","name":"tazcyr-12","timestamp":"2020-12-05T19:10:33.570642630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7GLmb4mMYo7YxTZc8acqrd3","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.575954_137.1","section":"bucket","name":"tazcyr-12","timestamp":"2020-12-05T19:10:33.575954766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7GLmb4mMYo7YxTZc8acqrd3","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.946968_148.1","section":"bucket.instance","name":"tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22","timestamp":"2020-12-05T19:10:54.946968611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pWRHIOzF5jjrCYjz7s-bT92","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.950994_149.1","section":"bucket.instance","name":"tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22","timestamp":"2020-12-05T19:10:54.950994376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pWRHIOzF5jjrCYjz7s-bT92","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.952799_150.1","section":"bucket","name":"tazcyr-13","timestamp":"2020-12-05T19:10:54.952799584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v1sOo4IxCDRkTTtx0nqQTQR","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.957698_151.1","section":"bucket","name":"tazcyr-13","timestamp":"2020-12-05T19:10:54.957698951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v1sOo4IxCDRkTTtx0nqQTQR","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.980569_152.1","section":"bucket.instance","name":"tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23","timestamp":"2020-12-05T19:10:54.980569878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0FBdQ8v781ipPDnrIzhREbU","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.985762_153.1","section":"bucket.instance","name":"tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23","timestamp":"2020-12-05T19:10:54.985762061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0FBdQ8v781ipPDnrIzhREbU","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.987699_154.1","section":"bucket","name":"tazcyr-14","timestamp":"2020-12-05T19:10:54.987699674Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ff-PcqwkIaDSj0MieJiq-nF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.991890_155.1","section":"bucket","name":"tazcyr-14","timestamp":"2020-12-05T19:10:54.991890859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ff-PcqwkIaDSj0MieJiq-nF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195455.033052_156.1","section":"bucket.instance","name":"tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24","timestamp":"2020-12-05T19:10:55.033052994Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rSc70PDLEverIcXCOefRxJC","ver":1},"status":{"status":"write"}}},{"id":"1_1607195455.038390_157.1","section":"bucket.instance","name":"tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24","timestamp":"2020-12-05T19:10:55.038390619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rSc70PDLEverIcXCOefRxJC","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195455.040782_158.1","section":"bucket","name":"tazcyr-15","timestamp":"2020-12-05T19:10:55.040782843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dHhEsmoErlujX90WT0hZ8b-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195455.044591_159.1","section":"bucket","name":"tazcyr-15","timestamp":"2020-12-05T19:10:55.044591823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dHhEsmoErlujX90WT0hZ8b-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.156008_180.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.156008202Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.160311_181.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.160311339Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.162280_182.1","section":"bucket","name":"tazcyr-16","timestamp":"2020-12-05T19:11:44.162280128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VJ5feBj8ie73TOSNm-mOk48","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.166249_183.1","section":"bucket","name":"tazcyr-16","timestamp":"2020-12-05T19:11:44.166249418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VJ5feBj8ie73TOSNm-mOk48","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.190323_184.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.190323520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.195436_185.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.195436227Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.197484_186.1","section":"bucket","name":"tazcyr-17","timestamp":"2020-12-05T19:11:44.197484394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A5Da6oy-XjJJ2vlamdSut1L","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.201637_187.1","section":"bucket","name":"tazcyr-17","timestamp":"2020-12-05T19:11:44.201637438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A5Da6oy-XjJJ2vlamdSut1L","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.245368_188.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.245368009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.249765_189.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.249765519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.251738_190.1","section":"bucket","name":"tazcyr-18","timestamp":"2020-12-05T19:11:44.251738627Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MLrGHAkNXQ9LQO7chnIbo9u","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.255952_191.1","section":"bucket","name":"tazcyr-18","timestamp":"2020-12-05T19:11:44.255952452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MLrGHAkNXQ9LQO7chnIbo9u","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.277957_192.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.277957424Z","data":{"read_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.283291_193.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.283291837Z","data":{"read_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195504.287523_194.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.287523427Z","data":{"read_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.293539_195.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.293539973Z","data":{"read_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195504.304166_196.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.304166482Z","data":{"read_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.311460_197.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.311460900Z","data":{"read_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195609.459177_203.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.459177746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"status":{"status":"write"}}},{"id":"1_1607195609.463517_204.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.463517024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195609.465743_205.1","section":"bucket","name":"tazcyr-19","timestamp":"2020-12-05T19:13:29.465743023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sqj9c_Y5K-PaIIhhHfp9aV-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195609.469886_206.1","section":"bucket","name":"tazcyr-19","timestamp":"2020-12-05T19:13:29.469886722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sqj9c_Y5K-PaIIhhHfp9aV-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195609.476273_207.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.476273667Z","data":{"read_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":2},"status":{"status":"write"}}},{"id":"1_1607195609.483992_208.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.483992218Z","data":{"read_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195673.656971_210.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:33.656971180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195673.662020_211.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:33.662020595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195673.664953_212.1","section":"bucket","name":"tazcyr-20","timestamp":"2020-12-05T19:14:33.664953183Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gbiHSoULZx5M93Ii9V39qaY","ver":1},"status":{"status":"write"}}},{"id":"1_1607195673.670005_213.1","section":"bucket","name":"tazcyr-20","timestamp":"2020-12-05T19:14:33.670005902Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gbiHSoULZx5M93Ii9V39qaY","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195693.006205_214.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:53.006205921Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"status":{"status":"write"}}},{"id":"1_1607195693.016790_215.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:53.016790934Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195713.362932_216.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:15:13.362932020Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":3},"status":{"status":"write"}}},{"id":"1_1607195713.373474_217.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:15:13.373474609Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195727.768234_218.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.768234903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.773284_219.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.773284550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.775253_220.1","section":"bucket","name":"tazcyr-21","timestamp":"2020-12-05T19:15:27.775253122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IiMD6iqfJAhr5nl3FFCo11h","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.778878_221.1","section":"bucket","name":"tazcyr-21","timestamp":"2020-12-05T19:15:27.778878626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IiMD6iqfJAhr5nl3FFCo11h","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.805082_222.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.805082256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.810030_223.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.810030473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.812453_224.1","section":"bucket","name":"tazcyr-22","timestamp":"2020-12-05T19:15:27.812453705Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vlaNvC_VGBzo9D9tFjShPpD","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.817658_225.1","section":"bucket","name":"tazcyr-22","timestamp":"2020-12-05T19:15:27.817658460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vlaNvC_VGBzo9D9tFjShPpD","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.872578_226.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.872578877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.877693_227.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.877693514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.880376_228.1","section":"bucket","name":"tazcyr-23","timestamp":"2020-12-05T19:15:27.880376747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1zdqPNwasiraGjdlhZp6oQt","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.887781_229.1","section":"bucket","name":"tazcyr-23","timestamp":"2020-12-05T19:15:27.887781658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1zdqPNwasiraGjdlhZp6oQt","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.910906_230.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.910906788Z","data":{"read_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.915864_231.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.915864011Z","data":{"read_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195727.920240_232.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.920240134Z","data":{"read_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.927455_233.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.927455161Z","data":{"read_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195727.938873_234.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.938873524Z","data":{"read_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.946934_235.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.946934371Z","data":{"read_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.202717_236.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.202717397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.206641_237.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.206641166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.208613_238.1","section":"bucket","name":"tazcyr-24","timestamp":"2020-12-05T19:15:58.208613089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJ9WH9704qJNfc87y_Uym40","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.212696_239.1","section":"bucket","name":"tazcyr-24","timestamp":"2020-12-05T19:15:58.212696835Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJ9WH9704qJNfc87y_Uym40","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.239556_240.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.239556477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.243770_241.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.243770434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.245921_242.1","section":"bucket","name":"tazcyr-25","timestamp":"2020-12-05T19:15:58.245921958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zxa0UHNz0xhSqsKfRDwukp8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.249853_243.1","section":"bucket","name":"tazcyr-25","timestamp":"2020-12-05T19:15:58.249853832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zxa0UHNz0xhSqsKfRDwukp8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.294853_244.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.294853051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.299247_245.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.299247495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.301130_246.1","section":"bucket","name":"tazcyr-26","timestamp":"2020-12-05T19:15:58.301130640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__lGAs5PGVMg5veawjx9eKO1","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.304797_247.1","section":"bucket","name":"tazcyr-26","timestamp":"2020-12-05T19:15:58.304797057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__lGAs5PGVMg5veawjx9eKO1","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.324771_248.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.324771799Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.329935_249.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.329935480Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.333372_250.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.333372026Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.339373_251.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.339373051Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195758.344276_252.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.344276591Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.350155_253.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.350155215Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.361818_254.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.361818610Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.369799_255.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.369799195Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195758.382775_256.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.382775320Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.391363_257.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.391363669Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.402710_258.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.402710892Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.409925_259.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.409925684Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195791.559884_260.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.559884544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.564310_261.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.564310563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.566185_262.1","section":"bucket","name":"tazcyr-27","timestamp":"2020-12-05T19:16:31.566185642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_JEiQXfZBSeXkBIrz6Wh64","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.570164_263.1","section":"bucket","name":"tazcyr-27","timestamp":"2020-12-05T19:16:31.570164527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_JEiQXfZBSeXkBIrz6Wh64","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.595904_264.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.595904056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.600782_265.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.600782574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.604311_266.1","section":"bucket","name":"tazcyr-28","timestamp":"2020-12-05T19:16:31.604311437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oc40Ilk1mlY412aky8o9il","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.611909_267.1","section":"bucket","name":"tazcyr-28","timestamp":"2020-12-05T19:16:31.611909175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oc40Ilk1mlY412aky8o9il","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.663356_268.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.663356474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.667691_269.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.667691372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.669956_270.1","section":"bucket","name":"tazcyr-29","timestamp":"2020-12-05T19:16:31.669956641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIrY6MlhiVSSiSqOOrUjlrR","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.674034_271.1","section":"bucket","name":"tazcyr-29","timestamp":"2020-12-05T19:16:31.674034946Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIrY6MlhiVSSiSqOOrUjlrR","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.698752_272.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.698752989Z","data":{"read_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.704822_273.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.704822499Z","data":{"read_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.751933_274.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.751933376Z","data":{"read_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.762446_275.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.762446815Z","data":{"read_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.823681_276.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.823681757Z","data":{"read_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.834360_277.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.834360396Z","data":{"read_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.910596_278.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:31.910596052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.916091_279.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:31.916091538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.918309_280.1","section":"bucket","name":"tazcyr-30","timestamp":"2020-12-05T19:16:31.918309043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V8JIXwA0EFqmAOaZ0sNYj58","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.922767_281.1","section":"bucket","name":"tazcyr-30","timestamp":"2020-12-05T19:16:31.922767856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V8JIXwA0EFqmAOaZ0sNYj58","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.952599_282.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:31.952599928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.956964_283.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:31.956964230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.958862_284.1","section":"bucket","name":"tazcyr-31","timestamp":"2020-12-05T19:16:31.958862465Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kBolvOrTYgZglOiG8VomhvM","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.962547_285.1","section":"bucket","name":"tazcyr-31","timestamp":"2020-12-05T19:16:31.962547883Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kBolvOrTYgZglOiG8VomhvM","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.015816_286.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.015816543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.020059_287.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.020059189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.021989_288.1","section":"bucket","name":"tazcyr-32","timestamp":"2020-12-05T19:16:32.021989230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-LCTSZbdQ2jP8bifQaJGDGi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.025960_289.1","section":"bucket","name":"tazcyr-32","timestamp":"2020-12-05T19:16:32.025960414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-LCTSZbdQ2jP8bifQaJGDGi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.049102_290.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:32.049102719Z","data":{"read_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.055546_291.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:32.055546086Z","data":{"read_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.105292_292.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:32.105292705Z","data":{"read_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.115903_293.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:32.115903395Z","data":{"read_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.173595_294.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.173595399Z","data":{"read_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.184392_295.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.184392397Z","data":{"read_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.261803_296.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.261803672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.266312_297.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.266312344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.268210_298.1","section":"bucket","name":"tazcyr-33","timestamp":"2020-12-05T19:16:32.268210820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVZLUD1vCqJ0Lt51HnkShsh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.273129_299.1","section":"bucket","name":"tazcyr-33","timestamp":"2020-12-05T19:16:32.273129813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVZLUD1vCqJ0Lt51HnkShsh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.300869_300.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:32.300869268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.306084_301.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:32.306084643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.308801_302.1","section":"bucket","name":"tazcyr-34","timestamp":"2020-12-05T19:16:32.308801370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iobgkLTDqhCewwIrJ3fJdW1","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.314464_303.1","section":"bucket","name":"tazcyr-34","timestamp":"2020-12-05T19:16:32.314464684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iobgkLTDqhCewwIrJ3fJdW1","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.370666_304.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:32.370666662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.377710_305.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:32.377710597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.379915_306.1","section":"bucket","name":"tazcyr-35","timestamp":"2020-12-05T19:16:32.379915477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lFts6vEJM81SxRXfGJh58Uh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.385846_307.1","section":"bucket","name":"tazcyr-35","timestamp":"2020-12-05T19:16:32.385846428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lFts6vEJM81SxRXfGJh58Uh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.412075_308.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.412075144Z","data":{"read_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.418632_309.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.418632526Z","data":{"read_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195793.468449_310.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:33.468449275Z","data":{"read_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":2},"status":{"status":"write"}}},{"id":"1_1607195793.480212_311.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:33.480212385Z","data":{"read_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195794.873408_312.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:34.873408400Z","data":{"read_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":2},"status":{"status":"write"}}},{"id":"1_1607195794.884443_313.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:34.884443336Z","data":{"read_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195795.745836_314.1","section":"bucket.instance","name":"tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45","timestamp":"2020-12-05T19:16:35.745836986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4ht6IvT3086xt5Cyjp-j0ZS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195795.750228_315.1","section":"bucket.instance","name":"tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45","timestamp":"2020-12-05T19:16:35.750228551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4ht6IvT3086xt5Cyjp-j0ZS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195795.752148_316.1","section":"bucket","name":"tazcyr-36","timestamp":"2020-12-05T19:16:35.752148872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oGGEcX5VQT-_WeBe3Thlum-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195795.756045_317.1","section":"bucket","name":"tazcyr-36","timestamp":"2020-12-05T19:16:35.756045544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oGGEcX5VQT-_WeBe3Thlum-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195796.395474_318.1","section":"bucket.instance","name":"tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46","timestamp":"2020-12-05T19:16:36.395474459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ynVeybecmoaDEfbRRNXnLxN","ver":1},"status":{"status":"write"}}},{"id":"1_1607195796.400062_319.1","section":"bucket.instance","name":"tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46","timestamp":"2020-12-05T19:16:36.400062414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ynVeybecmoaDEfbRRNXnLxN","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195796.402873_320.1","section":"bucket","name":"tazcyr-37","timestamp":"2020-12-05T19:16:36.402873361Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ETIrjIJ9hXuaj08WUt20_M","ver":1},"status":{"status":"write"}}},{"id":"1_1607195796.408510_321.1","section":"bucket","name":"tazcyr-37","timestamp":"2020-12-05T19:16:36.408510458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ETIrjIJ9hXuaj08WUt20_M","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195797.171740_322.1","section":"bucket.instance","name":"tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47","timestamp":"2020-12-05T19:16:37.171740685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ZQxJGge97W6bFOHGiwYws4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195797.177014_323.1","section":"bucket.instance","name":"tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47","timestamp":"2020-12-05T19:16:37.177014883Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ZQxJGge97W6bFOHGiwYws4","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195797.179628_324.1","section":"bucket","name":"tazcyr-38","timestamp":"2020-12-05T19:16:37.179628174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AamJhZR6s-WRjWe4vUMWqZr","ver":1},"status":{"status":"write"}}},{"id":"1_1607195797.184194_325.1","section":"bucket","name":"tazcyr-38","timestamp":"2020-12-05T19:16:37.184194689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AamJhZR6s-WRjWe4vUMWqZr","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195804.954889_326.1","section":"bucket.instance","name":"tazcyr-39:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48","timestamp":"2020-12-05T19:16:44.954889275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZhWn8Vk-C65I60Fnujl9kA8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195804.959965_327.1","section":"bucket.instance","name":"tazcyr-39:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48","timestamp":"2020-12-05T19:16:44.959965543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZhWn8Vk-C65I60Fnujl9kA8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195804.962048_328.1","section":"bucket","name":"tazcyr-39","timestamp":"2020-12-05T19:16:44.962048855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I5AIOg_F98j_OAJcf9wjuz-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195804.966788_329.1","section":"bucket","name":"tazcyr-39","timestamp":"2020-12-05T19:16:44.966788520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I5AIOg_F98j_OAJcf9wjuz-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.000660_330.1","section":"bucket.instance","name":"tazcyr-40:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49","timestamp":"2020-12-05T19:16:45.000660333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IN7Q23pS9W6mT4CaCf6VRbn","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.005998_331.1","section":"bucket.instance","name":"tazcyr-40:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49","timestamp":"2020-12-05T19:16:45.005998516Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IN7Q23pS9W6mT4CaCf6VRbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.008723_332.1","section":"bucket","name":"tazcyr-40","timestamp":"2020-12-05T19:16:45.008723804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VZMQhsTfCmn2-5GsJClxTBt","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.013709_333.1","section":"bucket","name":"tazcyr-40","timestamp":"2020-12-05T19:16:45.013709472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VZMQhsTfCmn2-5GsJClxTBt","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.063557_334.1","section":"bucket.instance","name":"tazcyr-41:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50","timestamp":"2020-12-05T19:16:45.063557672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KIlWf-4lWLe_zJjRcPBm7s3","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.068677_335.1","section":"bucket.instance","name":"tazcyr-41:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50","timestamp":"2020-12-05T19:16:45.068677879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KIlWf-4lWLe_zJjRcPBm7s3","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.071477_336.1","section":"bucket","name":"tazcyr-41","timestamp":"2020-12-05T19:16:45.071477845Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lXJLrvU6a2EnkXyJggSn-k4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.076974_337.1","section":"bucket","name":"tazcyr-41","timestamp":"2020-12-05T19:16:45.076974739Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lXJLrvU6a2EnkXyJggSn-k4","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:13.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:13.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:13.928 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period cb156068-23b3-4eca-8803-e19dbeb79445 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:13.968 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:13.969 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:14.251 INFO:teuthology.orchestra.run.smithi023.stdout:[{"id":"1_1607195257.607529_44.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-12-05T19:07:37.607529336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QzWxyJmSnbzgxXwE-SYm0Au","ver":1},"status":{"status":"write"}}},{"id":"1_1607195257.616573_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-12-05T19:07:37.616573496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QzWxyJmSnbzgxXwE-SYm0Au","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.004271_46.1","section":"bucket.instance","name":"tazcyr-1:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1","timestamp":"2020-12-05T19:09:40.004271326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzba4sBo5g6GI2dMc2uNhJS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.016937_47.1","section":"bucket.instance","name":"tazcyr-1:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1","timestamp":"2020-12-05T19:09:40.016937469Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzba4sBo5g6GI2dMc2uNhJS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.021417_48.1","section":"bucket","name":"tazcyr-1","timestamp":"2020-12-05T19:09:40.021417606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VyXMtA4g5xk2Opu-uU28KKS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.024544_49.1","section":"bucket","name":"tazcyr-1","timestamp":"2020-12-05T19:09:40.024544422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VyXMtA4g5xk2Opu-uU28KKS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.046042_50.1","section":"bucket.instance","name":"tazcyr-2:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2","timestamp":"2020-12-05T19:09:40.046042905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KP2FkRxiLEAiJe0MTCH9I4b","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.051202_51.1","section":"bucket.instance","name":"tazcyr-2:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2","timestamp":"2020-12-05T19:09:40.051202070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KP2FkRxiLEAiJe0MTCH9I4b","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.052778_52.1","section":"bucket","name":"tazcyr-2","timestamp":"2020-12-05T19:09:40.052778306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WUyHO6QQbtRgQbl6LIL5fmT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.057318_53.1","section":"bucket","name":"tazcyr-2","timestamp":"2020-12-05T19:09:40.057318427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WUyHO6QQbtRgQbl6LIL5fmT","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195383.710306_54.1","section":"bucket.instance","name":"tazcyr-3:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3","timestamp":"2020-12-05T19:09:43.710306623Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mYlUpB34g7c36jCJJY7xASF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195383.720265_55.1","section":"bucket.instance","name":"tazcyr-3:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3","timestamp":"2020-12-05T19:09:43.720265984Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mYlUpB34g7c36jCJJY7xASF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195383.722055_56.1","section":"bucket","name":"tazcyr-3","timestamp":"2020-12-05T19:09:43.722055553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YJJrKeypq2XGKHx5hyNMCIx","ver":1},"status":{"status":"write"}}},{"id":"1_1607195383.731707_57.1","section":"bucket","name":"tazcyr-3","timestamp":"2020-12-05T19:09:43.731707462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YJJrKeypq2XGKHx5hyNMCIx","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.335546_60.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4","timestamp":"2020-12-05T19:09:51.335546801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L_jTexV2QOKdHJrFnfn7kbh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.342055_61.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4","timestamp":"2020-12-05T19:09:51.342055564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L_jTexV2QOKdHJrFnfn7kbh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.344193_62.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:09:51.344193062Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jLnWUWkBmUYtUiCVFVQfp-b","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.351287_63.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:09:51.351287484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jLnWUWkBmUYtUiCVFVQfp-b","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.373162_64.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5","timestamp":"2020-12-05T19:09:51.373162509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E8T9igSLKPGjPwqoSjLEp_k","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.378587_65.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5","timestamp":"2020-12-05T19:09:51.378587248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E8T9igSLKPGjPwqoSjLEp_k","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.381243_66.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:09:51.381243284Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rz7EK0Cpc8qessr8O7Tn5iH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.388606_67.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:09:51.388606811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rz7EK0Cpc8qessr8O7Tn5iH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.437812_68.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6","timestamp":"2020-12-05T19:09:51.437812836Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZX-6V2b9MxDZwTcrSqix7n6","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.441771_69.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6","timestamp":"2020-12-05T19:09:51.441771244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZX-6V2b9MxDZwTcrSqix7n6","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.444005_70.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:09:51.444005575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6jyy8rxZVWt7mkAC_xVvmdF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.448654_71.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:09:51.448654007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6jyy8rxZVWt7mkAC_xVvmdF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.085465_72.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.7","timestamp":"2020-12-05T19:10:04.085465407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Cd-JV10QVaCV4xzrR7e8Cb","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.089157_73.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.7","timestamp":"2020-12-05T19:10:04.089157232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Cd-JV10QVaCV4xzrR7e8Cb","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.091040_74.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.091040641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ibQYDdoT09BZvl4zzz8MPh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.094388_75.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.094388936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ibQYDdoT09BZvl4zzz8MPh","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.124511_76.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.8","timestamp":"2020-12-05T19:10:04.124511324Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9u5vcAmfWRm3ybgEgHDxyJ7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.129450_77.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.8","timestamp":"2020-12-05T19:10:04.129450232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9u5vcAmfWRm3ybgEgHDxyJ7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.132112_78.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.132112638Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QhkUsfUtoco38o0vRSTiqJm","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.137475_79.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.137475791Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QhkUsfUtoco38o0vRSTiqJm","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.164520_80.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.9","timestamp":"2020-12-05T19:10:04.164520302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8G8bhkxk9kDUQA-JQCo8M0","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.169210_81.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.9","timestamp":"2020-12-05T19:10:04.169210675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8G8bhkxk9kDUQA-JQCo8M0","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.171105_82.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.171105195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ERy_2HXt0qrGTMck-HhMHoA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.174573_83.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.174573484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ERy_2HXt0qrGTMck-HhMHoA","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.202373_84.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.10","timestamp":"2020-12-05T19:10:04.202373262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjYrzFHFxksxKI8RLAch6d7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.206658_85.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.10","timestamp":"2020-12-05T19:10:04.206658084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjYrzFHFxksxKI8RLAch6d7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.209421_86.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.209421800Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_peuGzSEo8-wqUKsHfkgtZvP","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.212892_87.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.212892151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_peuGzSEo8-wqUKsHfkgtZvP","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.271570_88.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.11","timestamp":"2020-12-05T19:10:04.271570054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GIGteFOTnsUR-BSdJ7KuGyi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.276571_89.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.11","timestamp":"2020-12-05T19:10:04.276571979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GIGteFOTnsUR-BSdJ7KuGyi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.278943_90.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.278943293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZCB1N_YTX9IpoOGN96l67YT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.286241_91.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.286241492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZCB1N_YTX9IpoOGN96l67YT","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.393793_92.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.12","timestamp":"2020-12-05T19:10:04.393793199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wZ6ND_m9qn3va88zzyFK_lh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.398122_93.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.12","timestamp":"2020-12-05T19:10:04.398122874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wZ6ND_m9qn3va88zzyFK_lh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.400992_94.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.400992194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bvtob7IKitpBTl1g-h1fDWw","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.407785_95.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.407785961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bvtob7IKitpBTl1g-h1fDWw","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.516948_96.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.13","timestamp":"2020-12-05T19:10:04.516948552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_S9v1ugs3p0bAN19LWmAKo","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.521889_97.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.13","timestamp":"2020-12-05T19:10:04.521889148Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_S9v1ugs3p0bAN19LWmAKo","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.524060_98.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.524060551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRn-G8myGtlhg91Dk-Xg1zs","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.528641_99.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.528641393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRn-G8myGtlhg91Dk-Xg1zs","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.572841_100.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.14","timestamp":"2020-12-05T19:10:04.572841515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VKyqbVNoVDGlDXXzerwmsua","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.577361_101.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.14","timestamp":"2020-12-05T19:10:04.577361576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VKyqbVNoVDGlDXXzerwmsua","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.579161_102.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.579161032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_egt4Jp_vRt2MTLqMPIEdrsz","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.582561_103.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.582561935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_egt4Jp_vRt2MTLqMPIEdrsz","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.682129_104.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.15","timestamp":"2020-12-05T19:10:04.682129773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TiQoTfba0kD_Nn8vVWkQDe8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.689318_105.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.15","timestamp":"2020-12-05T19:10:04.689318302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TiQoTfba0kD_Nn8vVWkQDe8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.691755_106.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.691755301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJgnpoAQLxQuG357X5jW8Wb","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.697154_107.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.697154104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJgnpoAQLxQuG357X5jW8Wb","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195417.967417_108.1","section":"bucket.instance","name":"tazcyr-7:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16","timestamp":"2020-12-05T19:10:17.967417000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PK9gOINKQY1odbbT4wek8C_","ver":1},"status":{"status":"write"}}},{"id":"1_1607195417.971501_109.1","section":"bucket.instance","name":"tazcyr-7:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16","timestamp":"2020-12-05T19:10:17.971501874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PK9gOINKQY1odbbT4wek8C_","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195417.973539_110.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:17.973539330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195417.977586_111.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:17.977586121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195417.998346_112.1","section":"bucket.instance","name":"tazcyr-8:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17","timestamp":"2020-12-05T19:10:17.998346857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i7BUcpgTpgUsW-HrIAVrF6r","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.003566_113.1","section":"bucket.instance","name":"tazcyr-8:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17","timestamp":"2020-12-05T19:10:18.003566781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i7BUcpgTpgUsW-HrIAVrF6r","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.005584_114.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:18.005584098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.009519_115.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:18.009519950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.057689_116.1","section":"bucket.instance","name":"tazcyr-9:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18","timestamp":"2020-12-05T19:10:18.057689828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BCybjCVg7kHS2ZQlWPWBlz","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.061844_117.1","section":"bucket.instance","name":"tazcyr-9:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18","timestamp":"2020-12-05T19:10:18.061844526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BCybjCVg7kHS2ZQlWPWBlz","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.063969_118.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:18.063969600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.067947_119.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:18.067947008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195426.216787_120.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:26.216787778Z","data":{"read_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.221990_121.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:26.221990508Z","data":{"read_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195426.277724_122.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:26.277724745Z","data":{"read_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.284587_123.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:26.284587333Z","data":{"read_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195426.373298_124.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:26.373298757Z","data":{"read_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.378999_125.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:26.378999520Z","data":{"read_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195433.462730_126.1","section":"bucket.instance","name":"tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19","timestamp":"2020-12-05T19:10:33.462730080Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_52wD5vB-WvWwe7ywIABs_eP","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.467706_127.1","section":"bucket.instance","name":"tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19","timestamp":"2020-12-05T19:10:33.467706513Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_52wD5vB-WvWwe7ywIABs_eP","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.470216_128.1","section":"bucket","name":"tazcyr-10","timestamp":"2020-12-05T19:10:33.470216766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gF0E0avjnogGfUjbNKC363v","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.476060_129.1","section":"bucket","name":"tazcyr-10","timestamp":"2020-12-05T19:10:33.476060218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gF0E0avjnogGfUjbNKC363v","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.501303_130.1","section":"bucket.instance","name":"tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20","timestamp":"2020-12-05T19:10:33.501303520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IpmsnwQB3g2CXO9fV5tmzxe","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.505700_131.1","section":"bucket.instance","name":"tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20","timestamp":"2020-12-05T19:10:33.505700636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IpmsnwQB3g2CXO9fV5tmzxe","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.510386_132.1","section":"bucket","name":"tazcyr-11","timestamp":"2020-12-05T19:10:33.510386421Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCLh27ru7alDAFNCjpuYcbo","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.515084_133.1","section":"bucket","name":"tazcyr-11","timestamp":"2020-12-05T19:10:33.515084294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCLh27ru7alDAFNCjpuYcbo","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.563793_134.1","section":"bucket.instance","name":"tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21","timestamp":"2020-12-05T19:10:33.563793962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oEVFbZgDKIx5z8F7aheS42B","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.568424_135.1","section":"bucket.instance","name":"tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21","timestamp":"2020-12-05T19:10:33.568424303Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oEVFbZgDKIx5z8F7aheS42B","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.570642_136.1","section":"bucket","name":"tazcyr-12","timestamp":"2020-12-05T19:10:33.570642630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7GLmb4mMYo7YxTZc8acqrd3","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.575954_137.1","section":"bucket","name":"tazcyr-12","timestamp":"2020-12-05T19:10:33.575954766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7GLmb4mMYo7YxTZc8acqrd3","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.946968_148.1","section":"bucket.instance","name":"tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22","timestamp":"2020-12-05T19:10:54.946968611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pWRHIOzF5jjrCYjz7s-bT92","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.950994_149.1","section":"bucket.instance","name":"tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22","timestamp":"2020-12-05T19:10:54.950994376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pWRHIOzF5jjrCYjz7s-bT92","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.952799_150.1","section":"bucket","name":"tazcyr-13","timestamp":"2020-12-05T19:10:54.952799584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v1sOo4IxCDRkTTtx0nqQTQR","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.957698_151.1","section":"bucket","name":"tazcyr-13","timestamp":"2020-12-05T19:10:54.957698951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v1sOo4IxCDRkTTtx0nqQTQR","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.980569_152.1","section":"bucket.instance","name":"tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23","timestamp":"2020-12-05T19:10:54.980569878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0FBdQ8v781ipPDnrIzhREbU","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.985762_153.1","section":"bucket.instance","name":"tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23","timestamp":"2020-12-05T19:10:54.985762061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0FBdQ8v781ipPDnrIzhREbU","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.987699_154.1","section":"bucket","name":"tazcyr-14","timestamp":"2020-12-05T19:10:54.987699674Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ff-PcqwkIaDSj0MieJiq-nF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.991890_155.1","section":"bucket","name":"tazcyr-14","timestamp":"2020-12-05T19:10:54.991890859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ff-PcqwkIaDSj0MieJiq-nF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195455.033052_156.1","section":"bucket.instance","name":"tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24","timestamp":"2020-12-05T19:10:55.033052994Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rSc70PDLEverIcXCOefRxJC","ver":1},"status":{"status":"write"}}},{"id":"1_1607195455.038390_157.1","section":"bucket.instance","name":"tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24","timestamp":"2020-12-05T19:10:55.038390619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rSc70PDLEverIcXCOefRxJC","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195455.040782_158.1","section":"bucket","name":"tazcyr-15","timestamp":"2020-12-05T19:10:55.040782843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dHhEsmoErlujX90WT0hZ8b-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195455.044591_159.1","section":"bucket","name":"tazcyr-15","timestamp":"2020-12-05T19:10:55.044591823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dHhEsmoErlujX90WT0hZ8b-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.156008_180.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.156008202Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.160311_181.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.160311339Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.162280_182.1","section":"bucket","name":"tazcyr-16","timestamp":"2020-12-05T19:11:44.162280128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VJ5feBj8ie73TOSNm-mOk48","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.166249_183.1","section":"bucket","name":"tazcyr-16","timestamp":"2020-12-05T19:11:44.166249418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VJ5feBj8ie73TOSNm-mOk48","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.190323_184.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.190323520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.195436_185.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.195436227Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.197484_186.1","section":"bucket","name":"tazcyr-17","timestamp":"2020-12-05T19:11:44.197484394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A5Da6oy-XjJJ2vlamdSut1L","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.201637_187.1","section":"bucket","name":"tazcyr-17","timestamp":"2020-12-05T19:11:44.201637438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A5Da6oy-XjJJ2vlamdSut1L","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.245368_188.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.245368009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.249765_189.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.249765519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.251738_190.1","section":"bucket","name":"tazcyr-18","timestamp":"2020-12-05T19:11:44.251738627Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MLrGHAkNXQ9LQO7chnIbo9u","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.255952_191.1","section":"bucket","name":"tazcyr-18","timestamp":"2020-12-05T19:11:44.255952452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MLrGHAkNXQ9LQO7chnIbo9u","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.277957_192.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.277957424Z","data":{"read_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.283291_193.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.283291837Z","data":{"read_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195504.287523_194.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.287523427Z","data":{"read_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.293539_195.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.293539973Z","data":{"read_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195504.304166_196.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.304166482Z","data":{"read_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.311460_197.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.311460900Z","data":{"read_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195609.459177_203.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.459177746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"status":{"status":"write"}}},{"id":"1_1607195609.463517_204.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.463517024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195609.465743_205.1","section":"bucket","name":"tazcyr-19","timestamp":"2020-12-05T19:13:29.465743023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sqj9c_Y5K-PaIIhhHfp9aV-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195609.469886_206.1","section":"bucket","name":"tazcyr-19","timestamp":"2020-12-05T19:13:29.469886722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sqj9c_Y5K-PaIIhhHfp9aV-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195609.476273_207.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.476273667Z","data":{"read_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":2},"status":{"status":"write"}}},{"id":"1_1607195609.483992_208.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.483992218Z","data":{"read_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195673.656971_210.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:33.656971180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195673.662020_211.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:33.662020595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195673.664953_212.1","section":"bucket","name":"tazcyr-20","timestamp":"2020-12-05T19:14:33.664953183Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gbiHSoULZx5M93Ii9V39qaY","ver":1},"status":{"status":"write"}}},{"id":"1_1607195673.670005_213.1","section":"bucket","name":"tazcyr-20","timestamp":"2020-12-05T19:14:33.670005902Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gbiHSoULZx5M93Ii9V39qaY","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195693.006205_214.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:53.006205921Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"status":{"status":"write"}}},{"id":"1_1607195693.016790_215.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:53.016790934Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195713.362932_216.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:15:13.362932020Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":3},"status":{"status":"write"}}},{"id":"1_1607195713.373474_217.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:15:13.373474609Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195727.768234_218.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.768234903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.773284_219.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.773284550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.775253_220.1","section":"bucket","name":"tazcyr-21","timestamp":"2020-12-05T19:15:27.775253122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IiMD6iqfJAhr5nl3FFCo11h","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.778878_221.1","section":"bucket","name":"tazcyr-21","timestamp":"2020-12-05T19:15:27.778878626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IiMD6iqfJAhr5nl3FFCo11h","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.805082_222.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.805082256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.810030_223.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.810030473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.812453_224.1","section":"bucket","name":"tazcyr-22","timestamp":"2020-12-05T19:15:27.812453705Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vlaNvC_VGBzo9D9tFjShPpD","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.817658_225.1","section":"bucket","name":"tazcyr-22","timestamp":"2020-12-05T19:15:27.817658460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vlaNvC_VGBzo9D9tFjShPpD","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.872578_226.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.872578877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.877693_227.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.877693514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.880376_228.1","section":"bucket","name":"tazcyr-23","timestamp":"2020-12-05T19:15:27.880376747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1zdqPNwasiraGjdlhZp6oQt","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.887781_229.1","section":"bucket","name":"tazcyr-23","timestamp":"2020-12-05T19:15:27.887781658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1zdqPNwasiraGjdlhZp6oQt","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.910906_230.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.910906788Z","data":{"read_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.915864_231.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.915864011Z","data":{"read_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195727.920240_232.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.920240134Z","data":{"read_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.927455_233.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.927455161Z","data":{"read_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195727.938873_234.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.938873524Z","data":{"read_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.946934_235.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.946934371Z","data":{"read_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.202717_236.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.202717397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.206641_237.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.206641166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.208613_238.1","section":"bucket","name":"tazcyr-24","timestamp":"2020-12-05T19:15:58.208613089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJ9WH9704qJNfc87y_Uym40","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.212696_239.1","section":"bucket","name":"tazcyr-24","timestamp":"2020-12-05T19:15:58.212696835Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJ9WH9704qJNfc87y_Uym40","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.239556_240.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.239556477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.243770_241.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.243770434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.245921_242.1","section":"bucket","name":"tazcyr-25","timestamp":"2020-12-05T19:15:58.245921958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zxa0UHNz0xhSqsKfRDwukp8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.249853_243.1","section":"bucket","name":"tazcyr-25","timestamp":"2020-12-05T19:15:58.249853832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zxa0UHNz0xhSqsKfRDwukp8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.294853_244.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.294853051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.299247_245.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.299247495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.301130_246.1","section":"bucket","name":"tazcyr-26","timestamp":"2020-12-05T19:15:58.301130640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__lGAs5PGVMg5veawjx9eKO1","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.304797_247.1","section":"bucket","name":"tazcyr-26","timestamp":"2020-12-05T19:15:58.304797057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__lGAs5PGVMg5veawjx9eKO1","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.324771_248.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.324771799Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.329935_249.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.329935480Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.333372_250.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.333372026Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.339373_251.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.339373051Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195758.344276_252.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.344276591Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.350155_253.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.350155215Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.361818_254.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.361818610Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.369799_255.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.369799195Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195758.382775_256.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.382775320Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.391363_257.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.391363669Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.402710_258.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.402710892Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.409925_259.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.409925684Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195791.559884_260.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.559884544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.564310_261.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.564310563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.566185_262.1","section":"bucket","name":"tazcyr-27","timestamp":"2020-12-05T19:16:31.566185642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_JEiQXfZBSeXkBIrz6Wh64","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.570164_263.1","section":"bucket","name":"tazcyr-27","timestamp":"2020-12-05T19:16:31.570164527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_JEiQXfZBSeXkBIrz6Wh64","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.595904_264.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.595904056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.600782_265.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.600782574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.604311_266.1","section":"bucket","name":"tazcyr-28","timestamp":"2020-12-05T19:16:31.604311437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oc40Ilk1mlY412aky8o9il","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.611909_267.1","section":"bucket","name":"tazcyr-28","timestamp":"2020-12-05T19:16:31.611909175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oc40Ilk1mlY412aky8o9il","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.663356_268.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.663356474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.667691_269.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.667691372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.669956_270.1","section":"bucket","name":"tazcyr-29","timestamp":"2020-12-05T19:16:31.669956641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIrY6MlhiVSSiSqOOrUjlrR","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.674034_271.1","section":"bucket","name":"tazcyr-29","timestamp":"2020-12-05T19:16:31.674034946Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIrY6MlhiVSSiSqOOrUjlrR","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.698752_272.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.698752989Z","data":{"read_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.704822_273.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.704822499Z","data":{"read_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.751933_274.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.751933376Z","data":{"read_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.762446_275.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.762446815Z","data":{"read_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.823681_276.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.823681757Z","data":{"read_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.834360_277.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.834360396Z","data":{"read_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.910596_278.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:31.910596052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.916091_279.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:31.916091538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.918309_280.1","section":"bucket","name":"tazcyr-30","timestamp":"2020-12-05T19:16:31.918309043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V8JIXwA0EFqmAOaZ0sNYj58","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.922767_281.1","section":"bucket","name":"tazcyr-30","timestamp":"2020-12-05T19:16:31.922767856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V8JIXwA0EFqmAOaZ0sNYj58","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.952599_282.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:31.952599928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.956964_283.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:31.956964230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.958862_284.1","section":"bucket","name":"tazcyr-31","timestamp":"2020-12-05T19:16:31.958862465Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kBolvOrTYgZglOiG8VomhvM","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.962547_285.1","section":"bucket","name":"tazcyr-31","timestamp":"2020-12-05T19:16:31.962547883Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kBolvOrTYgZglOiG8VomhvM","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.015816_286.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.015816543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.020059_287.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.020059189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.021989_288.1","section":"bucket","name":"tazcyr-32","timestamp":"2020-12-05T19:16:32.021989230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-LCTSZbdQ2jP8bifQaJGDGi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.025960_289.1","section":"bucket","name":"tazcyr-32","timestamp":"2020-12-05T19:16:32.025960414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-LCTSZbdQ2jP8bifQaJGDGi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.049102_290.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:32.049102719Z","data":{"read_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.055546_291.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:32.055546086Z","data":{"read_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.105292_292.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:32.105292705Z","data":{"read_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.115903_293.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:32.115903395Z","data":{"read_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.173595_294.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.173595399Z","data":{"read_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.184392_295.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.184392397Z","data":{"read_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.261803_296.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.261803672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.266312_297.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.266312344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.268210_298.1","section":"bucket","name":"tazcyr-33","timestamp":"2020-12-05T19:16:32.268210820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVZLUD1vCqJ0Lt51HnkShsh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.273129_299.1","section":"bucket","name":"tazcyr-33","timestamp":"2020-12-05T19:16:32.273129813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVZLUD1vCqJ0Lt51HnkShsh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.300869_300.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:32.300869268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.306084_301.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:32.306084643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.308801_302.1","section":"bucket","name":"tazcyr-34","timestamp":"2020-12-05T19:16:32.308801370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iobgkLTDqhCewwIrJ3fJdW1","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.314464_303.1","section":"bucket","name":"tazcyr-34","timestamp":"2020-12-05T19:16:32.314464684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iobgkLTDqhCewwIrJ3fJdW1","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.370666_304.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:32.370666662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.377710_305.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:32.377710597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.379915_306.1","section":"bucket","name":"tazcyr-35","timestamp":"2020-12-05T19:16:32.379915477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lFts6vEJM81SxRXfGJh58Uh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.385846_307.1","section":"bucket","name":"tazcyr-35","timestamp":"2020-12-05T19:16:32.385846428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lFts6vEJM81SxRXfGJh58Uh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.412075_308.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.412075144Z","data":{"read_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.418632_309.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.418632526Z","data":{"read_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195793.468449_310.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:33.468449275Z","data":{"read_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":2},"status":{"status":"write"}}},{"id":"1_1607195793.480212_311.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:33.480212385Z","data":{"read_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195794.873408_312.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:34.873408400Z","data":{"read_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":2},"status":{"status":"write"}}},{"id":"1_1607195794.884443_313.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:34.884443336Z","data":{"read_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195795.745836_314.1","section":"bucket.instance","name":"tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45","timestamp":"2020-12-05T19:16:35.745836986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4ht6IvT3086xt5Cyjp-j0ZS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195795.750228_315.1","section":"bucket.instance","name":"tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45","timestamp":"2020-12-05T19:16:35.750228551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4ht6IvT3086xt5Cyjp-j0ZS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195795.752148_316.1","section":"bucket","name":"tazcyr-36","timestamp":"2020-12-05T19:16:35.752148872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oGGEcX5VQT-_WeBe3Thlum-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195795.756045_317.1","section":"bucket","name":"tazcyr-36","timestamp":"2020-12-05T19:16:35.756045544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oGGEcX5VQT-_WeBe3Thlum-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195796.395474_318.1","section":"bucket.instance","name":"tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46","timestamp":"2020-12-05T19:16:36.395474459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ynVeybecmoaDEfbRRNXnLxN","ver":1},"status":{"status":"write"}}},{"id":"1_1607195796.400062_319.1","section":"bucket.instance","name":"tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46","timestamp":"2020-12-05T19:16:36.400062414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ynVeybecmoaDEfbRRNXnLxN","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195796.402873_320.1","section":"bucket","name":"tazcyr-37","timestamp":"2020-12-05T19:16:36.402873361Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ETIrjIJ9hXuaj08WUt20_M","ver":1},"status":{"status":"write"}}},{"id":"1_1607195796.408510_321.1","section":"bucket","name":"tazcyr-37","timestamp":"2020-12-05T19:16:36.408510458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ETIrjIJ9hXuaj08WUt20_M","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195797.171740_322.1","section":"bucket.instance","name":"tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47","timestamp":"2020-12-05T19:16:37.171740685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ZQxJGge97W6bFOHGiwYws4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195797.177014_323.1","section":"bucket.instance","name":"tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47","timestamp":"2020-12-05T19:16:37.177014883Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ZQxJGge97W6bFOHGiwYws4","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195797.179628_324.1","section":"bucket","name":"tazcyr-38","timestamp":"2020-12-05T19:16:37.179628174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AamJhZR6s-WRjWe4vUMWqZr","ver":1},"status":{"status":"write"}}},{"id":"1_1607195797.184194_325.1","section":"bucket","name":"tazcyr-38","timestamp":"2020-12-05T19:16:37.184194689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AamJhZR6s-WRjWe4vUMWqZr","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195804.954889_326.1","section":"bucket.instance","name":"tazcyr-39:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48","timestamp":"2020-12-05T19:16:44.954889275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZhWn8Vk-C65I60Fnujl9kA8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195804.959965_327.1","section":"bucket.instance","name":"tazcyr-39:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48","timestamp":"2020-12-05T19:16:44.959965543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZhWn8Vk-C65I60Fnujl9kA8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195804.962048_328.1","section":"bucket","name":"tazcyr-39","timestamp":"2020-12-05T19:16:44.962048855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I5AIOg_F98j_OAJcf9wjuz-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195804.966788_329.1","section":"bucket","name":"tazcyr-39","timestamp":"2020-12-05T19:16:44.966788520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I5AIOg_F98j_OAJcf9wjuz-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.000660_330.1","section":"bucket.instance","name":"tazcyr-40:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49","timestamp":"2020-12-05T19:16:45.000660333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IN7Q23pS9W6mT4CaCf6VRbn","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.005998_331.1","section":"bucket.instance","name":"tazcyr-40:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49","timestamp":"2020-12-05T19:16:45.005998516Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IN7Q23pS9W6mT4CaCf6VRbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.008723_332.1","section":"bucket","name":"tazcyr-40","timestamp":"2020-12-05T19:16:45.008723804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VZMQhsTfCmn2-5GsJClxTBt","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.013709_333.1","section":"bucket","name":"tazcyr-40","timestamp":"2020-12-05T19:16:45.013709472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VZMQhsTfCmn2-5GsJClxTBt","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.063557_334.1","section":"bucket.instance","name":"tazcyr-41:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50","timestamp":"2020-12-05T19:16:45.063557672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KIlWf-4lWLe_zJjRcPBm7s3","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.068677_335.1","section":"bucket.instance","name":"tazcyr-41:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50","timestamp":"2020-12-05T19:16:45.068677879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KIlWf-4lWLe_zJjRcPBm7s3","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.071477_336.1","section":"bucket","name":"tazcyr-41","timestamp":"2020-12-05T19:16:45.071477845Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lXJLrvU6a2EnkXyJggSn-k4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.076974_337.1","section":"bucket","name":"tazcyr-41","timestamp":"2020-12-05T19:16:45.076974739Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lXJLrvU6a2EnkXyJggSn-k4","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:14.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:14.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:14.254 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period cb156068-23b3-4eca-8803-e19dbeb79445 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:14.291 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:14.291 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:14.586 INFO:teuthology.orchestra.run.smithi185.stdout:[{"id":"1_1607195257.607529_44.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-12-05T19:07:37.607529336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QzWxyJmSnbzgxXwE-SYm0Au","ver":1},"status":{"status":"write"}}},{"id":"1_1607195257.616573_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-12-05T19:07:37.616573496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QzWxyJmSnbzgxXwE-SYm0Au","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.004271_46.1","section":"bucket.instance","name":"tazcyr-1:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1","timestamp":"2020-12-05T19:09:40.004271326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzba4sBo5g6GI2dMc2uNhJS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.016937_47.1","section":"bucket.instance","name":"tazcyr-1:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1","timestamp":"2020-12-05T19:09:40.016937469Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzba4sBo5g6GI2dMc2uNhJS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.021417_48.1","section":"bucket","name":"tazcyr-1","timestamp":"2020-12-05T19:09:40.021417606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VyXMtA4g5xk2Opu-uU28KKS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.024544_49.1","section":"bucket","name":"tazcyr-1","timestamp":"2020-12-05T19:09:40.024544422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VyXMtA4g5xk2Opu-uU28KKS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.046042_50.1","section":"bucket.instance","name":"tazcyr-2:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2","timestamp":"2020-12-05T19:09:40.046042905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KP2FkRxiLEAiJe0MTCH9I4b","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.051202_51.1","section":"bucket.instance","name":"tazcyr-2:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2","timestamp":"2020-12-05T19:09:40.051202070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KP2FkRxiLEAiJe0MTCH9I4b","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.052778_52.1","section":"bucket","name":"tazcyr-2","timestamp":"2020-12-05T19:09:40.052778306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WUyHO6QQbtRgQbl6LIL5fmT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.057318_53.1","section":"bucket","name":"tazcyr-2","timestamp":"2020-12-05T19:09:40.057318427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WUyHO6QQbtRgQbl6LIL5fmT","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195383.710306_54.1","section":"bucket.instance","name":"tazcyr-3:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3","timestamp":"2020-12-05T19:09:43.710306623Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mYlUpB34g7c36jCJJY7xASF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195383.720265_55.1","section":"bucket.instance","name":"tazcyr-3:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3","timestamp":"2020-12-05T19:09:43.720265984Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mYlUpB34g7c36jCJJY7xASF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195383.722055_56.1","section":"bucket","name":"tazcyr-3","timestamp":"2020-12-05T19:09:43.722055553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YJJrKeypq2XGKHx5hyNMCIx","ver":1},"status":{"status":"write"}}},{"id":"1_1607195383.731707_57.1","section":"bucket","name":"tazcyr-3","timestamp":"2020-12-05T19:09:43.731707462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YJJrKeypq2XGKHx5hyNMCIx","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.335546_60.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4","timestamp":"2020-12-05T19:09:51.335546801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L_jTexV2QOKdHJrFnfn7kbh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.342055_61.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4","timestamp":"2020-12-05T19:09:51.342055564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L_jTexV2QOKdHJrFnfn7kbh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.344193_62.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:09:51.344193062Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jLnWUWkBmUYtUiCVFVQfp-b","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.351287_63.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:09:51.351287484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jLnWUWkBmUYtUiCVFVQfp-b","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.373162_64.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5","timestamp":"2020-12-05T19:09:51.373162509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E8T9igSLKPGjPwqoSjLEp_k","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.378587_65.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5","timestamp":"2020-12-05T19:09:51.378587248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E8T9igSLKPGjPwqoSjLEp_k","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.381243_66.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:09:51.381243284Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rz7EK0Cpc8qessr8O7Tn5iH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.388606_67.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:09:51.388606811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rz7EK0Cpc8qessr8O7Tn5iH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.437812_68.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6","timestamp":"2020-12-05T19:09:51.437812836Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZX-6V2b9MxDZwTcrSqix7n6","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.441771_69.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6","timestamp":"2020-12-05T19:09:51.441771244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZX-6V2b9MxDZwTcrSqix7n6","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.444005_70.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:09:51.444005575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6jyy8rxZVWt7mkAC_xVvmdF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.448654_71.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:09:51.448654007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6jyy8rxZVWt7mkAC_xVvmdF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.085465_72.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.7","timestamp":"2020-12-05T19:10:04.085465407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Cd-JV10QVaCV4xzrR7e8Cb","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.089157_73.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.7","timestamp":"2020-12-05T19:10:04.089157232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Cd-JV10QVaCV4xzrR7e8Cb","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.091040_74.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.091040641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ibQYDdoT09BZvl4zzz8MPh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.094388_75.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.094388936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ibQYDdoT09BZvl4zzz8MPh","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.124511_76.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.8","timestamp":"2020-12-05T19:10:04.124511324Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9u5vcAmfWRm3ybgEgHDxyJ7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.129450_77.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.8","timestamp":"2020-12-05T19:10:04.129450232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9u5vcAmfWRm3ybgEgHDxyJ7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.132112_78.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.132112638Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QhkUsfUtoco38o0vRSTiqJm","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.137475_79.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.137475791Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QhkUsfUtoco38o0vRSTiqJm","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.164520_80.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.9","timestamp":"2020-12-05T19:10:04.164520302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8G8bhkxk9kDUQA-JQCo8M0","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.169210_81.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.9","timestamp":"2020-12-05T19:10:04.169210675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8G8bhkxk9kDUQA-JQCo8M0","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.171105_82.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.171105195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ERy_2HXt0qrGTMck-HhMHoA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.174573_83.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.174573484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ERy_2HXt0qrGTMck-HhMHoA","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.202373_84.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.10","timestamp":"2020-12-05T19:10:04.202373262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjYrzFHFxksxKI8RLAch6d7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.206658_85.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.10","timestamp":"2020-12-05T19:10:04.206658084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjYrzFHFxksxKI8RLAch6d7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.209421_86.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.209421800Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_peuGzSEo8-wqUKsHfkgtZvP","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.212892_87.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.212892151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_peuGzSEo8-wqUKsHfkgtZvP","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.271570_88.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.11","timestamp":"2020-12-05T19:10:04.271570054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GIGteFOTnsUR-BSdJ7KuGyi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.276571_89.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.11","timestamp":"2020-12-05T19:10:04.276571979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GIGteFOTnsUR-BSdJ7KuGyi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.278943_90.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.278943293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZCB1N_YTX9IpoOGN96l67YT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.286241_91.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.286241492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZCB1N_YTX9IpoOGN96l67YT","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.393793_92.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.12","timestamp":"2020-12-05T19:10:04.393793199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wZ6ND_m9qn3va88zzyFK_lh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.398122_93.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.12","timestamp":"2020-12-05T19:10:04.398122874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wZ6ND_m9qn3va88zzyFK_lh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.400992_94.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.400992194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bvtob7IKitpBTl1g-h1fDWw","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.407785_95.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.407785961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bvtob7IKitpBTl1g-h1fDWw","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.516948_96.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.13","timestamp":"2020-12-05T19:10:04.516948552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_S9v1ugs3p0bAN19LWmAKo","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.521889_97.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.13","timestamp":"2020-12-05T19:10:04.521889148Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_S9v1ugs3p0bAN19LWmAKo","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.524060_98.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.524060551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRn-G8myGtlhg91Dk-Xg1zs","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.528641_99.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.528641393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRn-G8myGtlhg91Dk-Xg1zs","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.572841_100.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.14","timestamp":"2020-12-05T19:10:04.572841515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VKyqbVNoVDGlDXXzerwmsua","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.577361_101.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.14","timestamp":"2020-12-05T19:10:04.577361576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VKyqbVNoVDGlDXXzerwmsua","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.579161_102.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.579161032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_egt4Jp_vRt2MTLqMPIEdrsz","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.582561_103.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.582561935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_egt4Jp_vRt2MTLqMPIEdrsz","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.682129_104.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.15","timestamp":"2020-12-05T19:10:04.682129773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TiQoTfba0kD_Nn8vVWkQDe8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.689318_105.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.15","timestamp":"2020-12-05T19:10:04.689318302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TiQoTfba0kD_Nn8vVWkQDe8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.691755_106.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.691755301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJgnpoAQLxQuG357X5jW8Wb","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.697154_107.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.697154104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJgnpoAQLxQuG357X5jW8Wb","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195417.967417_108.1","section":"bucket.instance","name":"tazcyr-7:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16","timestamp":"2020-12-05T19:10:17.967417000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PK9gOINKQY1odbbT4wek8C_","ver":1},"status":{"status":"write"}}},{"id":"1_1607195417.971501_109.1","section":"bucket.instance","name":"tazcyr-7:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16","timestamp":"2020-12-05T19:10:17.971501874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PK9gOINKQY1odbbT4wek8C_","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195417.973539_110.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:17.973539330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195417.977586_111.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:17.977586121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195417.998346_112.1","section":"bucket.instance","name":"tazcyr-8:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17","timestamp":"2020-12-05T19:10:17.998346857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i7BUcpgTpgUsW-HrIAVrF6r","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.003566_113.1","section":"bucket.instance","name":"tazcyr-8:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17","timestamp":"2020-12-05T19:10:18.003566781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i7BUcpgTpgUsW-HrIAVrF6r","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.005584_114.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:18.005584098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.009519_115.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:18.009519950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.057689_116.1","section":"bucket.instance","name":"tazcyr-9:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18","timestamp":"2020-12-05T19:10:18.057689828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BCybjCVg7kHS2ZQlWPWBlz","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.061844_117.1","section":"bucket.instance","name":"tazcyr-9:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18","timestamp":"2020-12-05T19:10:18.061844526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BCybjCVg7kHS2ZQlWPWBlz","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.063969_118.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:18.063969600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.067947_119.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:18.067947008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195426.216787_120.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:26.216787778Z","data":{"read_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.221990_121.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:26.221990508Z","data":{"read_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195426.277724_122.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:26.277724745Z","data":{"read_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.284587_123.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:26.284587333Z","data":{"read_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195426.373298_124.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:26.373298757Z","data":{"read_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.378999_125.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:26.378999520Z","data":{"read_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195433.462730_126.1","section":"bucket.instance","name":"tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19","timestamp":"2020-12-05T19:10:33.462730080Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_52wD5vB-WvWwe7ywIABs_eP","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.467706_127.1","section":"bucket.instance","name":"tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19","timestamp":"2020-12-05T19:10:33.467706513Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_52wD5vB-WvWwe7ywIABs_eP","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.470216_128.1","section":"bucket","name":"tazcyr-10","timestamp":"2020-12-05T19:10:33.470216766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gF0E0avjnogGfUjbNKC363v","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.476060_129.1","section":"bucket","name":"tazcyr-10","timestamp":"2020-12-05T19:10:33.476060218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gF0E0avjnogGfUjbNKC363v","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.501303_130.1","section":"bucket.instance","name":"tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20","timestamp":"2020-12-05T19:10:33.501303520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IpmsnwQB3g2CXO9fV5tmzxe","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.505700_131.1","section":"bucket.instance","name":"tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20","timestamp":"2020-12-05T19:10:33.505700636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IpmsnwQB3g2CXO9fV5tmzxe","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.510386_132.1","section":"bucket","name":"tazcyr-11","timestamp":"2020-12-05T19:10:33.510386421Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCLh27ru7alDAFNCjpuYcbo","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.515084_133.1","section":"bucket","name":"tazcyr-11","timestamp":"2020-12-05T19:10:33.515084294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCLh27ru7alDAFNCjpuYcbo","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.563793_134.1","section":"bucket.instance","name":"tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21","timestamp":"2020-12-05T19:10:33.563793962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oEVFbZgDKIx5z8F7aheS42B","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.568424_135.1","section":"bucket.instance","name":"tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21","timestamp":"2020-12-05T19:10:33.568424303Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oEVFbZgDKIx5z8F7aheS42B","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.570642_136.1","section":"bucket","name":"tazcyr-12","timestamp":"2020-12-05T19:10:33.570642630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7GLmb4mMYo7YxTZc8acqrd3","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.575954_137.1","section":"bucket","name":"tazcyr-12","timestamp":"2020-12-05T19:10:33.575954766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7GLmb4mMYo7YxTZc8acqrd3","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.946968_148.1","section":"bucket.instance","name":"tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22","timestamp":"2020-12-05T19:10:54.946968611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pWRHIOzF5jjrCYjz7s-bT92","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.950994_149.1","section":"bucket.instance","name":"tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22","timestamp":"2020-12-05T19:10:54.950994376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pWRHIOzF5jjrCYjz7s-bT92","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.952799_150.1","section":"bucket","name":"tazcyr-13","timestamp":"2020-12-05T19:10:54.952799584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v1sOo4IxCDRkTTtx0nqQTQR","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.957698_151.1","section":"bucket","name":"tazcyr-13","timestamp":"2020-12-05T19:10:54.957698951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v1sOo4IxCDRkTTtx0nqQTQR","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.980569_152.1","section":"bucket.instance","name":"tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23","timestamp":"2020-12-05T19:10:54.980569878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0FBdQ8v781ipPDnrIzhREbU","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.985762_153.1","section":"bucket.instance","name":"tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23","timestamp":"2020-12-05T19:10:54.985762061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0FBdQ8v781ipPDnrIzhREbU","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.987699_154.1","section":"bucket","name":"tazcyr-14","timestamp":"2020-12-05T19:10:54.987699674Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ff-PcqwkIaDSj0MieJiq-nF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.991890_155.1","section":"bucket","name":"tazcyr-14","timestamp":"2020-12-05T19:10:54.991890859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ff-PcqwkIaDSj0MieJiq-nF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195455.033052_156.1","section":"bucket.instance","name":"tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24","timestamp":"2020-12-05T19:10:55.033052994Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rSc70PDLEverIcXCOefRxJC","ver":1},"status":{"status":"write"}}},{"id":"1_1607195455.038390_157.1","section":"bucket.instance","name":"tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24","timestamp":"2020-12-05T19:10:55.038390619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rSc70PDLEverIcXCOefRxJC","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195455.040782_158.1","section":"bucket","name":"tazcyr-15","timestamp":"2020-12-05T19:10:55.040782843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dHhEsmoErlujX90WT0hZ8b-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195455.044591_159.1","section":"bucket","name":"tazcyr-15","timestamp":"2020-12-05T19:10:55.044591823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dHhEsmoErlujX90WT0hZ8b-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.156008_180.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.156008202Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.160311_181.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.160311339Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.162280_182.1","section":"bucket","name":"tazcyr-16","timestamp":"2020-12-05T19:11:44.162280128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VJ5feBj8ie73TOSNm-mOk48","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.166249_183.1","section":"bucket","name":"tazcyr-16","timestamp":"2020-12-05T19:11:44.166249418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VJ5feBj8ie73TOSNm-mOk48","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.190323_184.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.190323520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.195436_185.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.195436227Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.197484_186.1","section":"bucket","name":"tazcyr-17","timestamp":"2020-12-05T19:11:44.197484394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A5Da6oy-XjJJ2vlamdSut1L","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.201637_187.1","section":"bucket","name":"tazcyr-17","timestamp":"2020-12-05T19:11:44.201637438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A5Da6oy-XjJJ2vlamdSut1L","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.245368_188.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.245368009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.249765_189.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.249765519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.251738_190.1","section":"bucket","name":"tazcyr-18","timestamp":"2020-12-05T19:11:44.251738627Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MLrGHAkNXQ9LQO7chnIbo9u","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.255952_191.1","section":"bucket","name":"tazcyr-18","timestamp":"2020-12-05T19:11:44.255952452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MLrGHAkNXQ9LQO7chnIbo9u","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.277957_192.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.277957424Z","data":{"read_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.283291_193.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.283291837Z","data":{"read_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195504.287523_194.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.287523427Z","data":{"read_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.293539_195.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.293539973Z","data":{"read_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195504.304166_196.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.304166482Z","data":{"read_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.311460_197.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.311460900Z","data":{"read_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195609.459177_203.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.459177746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"status":{"status":"write"}}},{"id":"1_1607195609.463517_204.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.463517024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195609.465743_205.1","section":"bucket","name":"tazcyr-19","timestamp":"2020-12-05T19:13:29.465743023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sqj9c_Y5K-PaIIhhHfp9aV-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195609.469886_206.1","section":"bucket","name":"tazcyr-19","timestamp":"2020-12-05T19:13:29.469886722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sqj9c_Y5K-PaIIhhHfp9aV-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195609.476273_207.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.476273667Z","data":{"read_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":2},"status":{"status":"write"}}},{"id":"1_1607195609.483992_208.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.483992218Z","data":{"read_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195673.656971_210.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:33.656971180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195673.662020_211.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:33.662020595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195673.664953_212.1","section":"bucket","name":"tazcyr-20","timestamp":"2020-12-05T19:14:33.664953183Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gbiHSoULZx5M93Ii9V39qaY","ver":1},"status":{"status":"write"}}},{"id":"1_1607195673.670005_213.1","section":"bucket","name":"tazcyr-20","timestamp":"2020-12-05T19:14:33.670005902Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gbiHSoULZx5M93Ii9V39qaY","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195693.006205_214.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:53.006205921Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"status":{"status":"write"}}},{"id":"1_1607195693.016790_215.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:53.016790934Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195713.362932_216.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:15:13.362932020Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":3},"status":{"status":"write"}}},{"id":"1_1607195713.373474_217.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:15:13.373474609Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195727.768234_218.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.768234903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.773284_219.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.773284550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.775253_220.1","section":"bucket","name":"tazcyr-21","timestamp":"2020-12-05T19:15:27.775253122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IiMD6iqfJAhr5nl3FFCo11h","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.778878_221.1","section":"bucket","name":"tazcyr-21","timestamp":"2020-12-05T19:15:27.778878626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IiMD6iqfJAhr5nl3FFCo11h","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.805082_222.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.805082256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.810030_223.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.810030473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.812453_224.1","section":"bucket","name":"tazcyr-22","timestamp":"2020-12-05T19:15:27.812453705Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vlaNvC_VGBzo9D9tFjShPpD","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.817658_225.1","section":"bucket","name":"tazcyr-22","timestamp":"2020-12-05T19:15:27.817658460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vlaNvC_VGBzo9D9tFjShPpD","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.872578_226.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.872578877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.877693_227.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.877693514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.880376_228.1","section":"bucket","name":"tazcyr-23","timestamp":"2020-12-05T19:15:27.880376747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1zdqPNwasiraGjdlhZp6oQt","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.887781_229.1","section":"bucket","name":"tazcyr-23","timestamp":"2020-12-05T19:15:27.887781658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1zdqPNwasiraGjdlhZp6oQt","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.910906_230.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.910906788Z","data":{"read_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.915864_231.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.915864011Z","data":{"read_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195727.920240_232.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.920240134Z","data":{"read_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.927455_233.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.927455161Z","data":{"read_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195727.938873_234.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.938873524Z","data":{"read_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.946934_235.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.946934371Z","data":{"read_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.202717_236.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.202717397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.206641_237.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.206641166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.208613_238.1","section":"bucket","name":"tazcyr-24","timestamp":"2020-12-05T19:15:58.208613089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJ9WH9704qJNfc87y_Uym40","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.212696_239.1","section":"bucket","name":"tazcyr-24","timestamp":"2020-12-05T19:15:58.212696835Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJ9WH9704qJNfc87y_Uym40","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.239556_240.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.239556477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.243770_241.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.243770434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.245921_242.1","section":"bucket","name":"tazcyr-25","timestamp":"2020-12-05T19:15:58.245921958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zxa0UHNz0xhSqsKfRDwukp8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.249853_243.1","section":"bucket","name":"tazcyr-25","timestamp":"2020-12-05T19:15:58.249853832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zxa0UHNz0xhSqsKfRDwukp8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.294853_244.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.294853051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.299247_245.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.299247495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.301130_246.1","section":"bucket","name":"tazcyr-26","timestamp":"2020-12-05T19:15:58.301130640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__lGAs5PGVMg5veawjx9eKO1","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.304797_247.1","section":"bucket","name":"tazcyr-26","timestamp":"2020-12-05T19:15:58.304797057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__lGAs5PGVMg5veawjx9eKO1","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.324771_248.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.324771799Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.329935_249.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.329935480Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.333372_250.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.333372026Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.339373_251.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.339373051Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195758.344276_252.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.344276591Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.350155_253.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.350155215Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.361818_254.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.361818610Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.369799_255.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.369799195Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195758.382775_256.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.382775320Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.391363_257.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.391363669Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.402710_258.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.402710892Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.409925_259.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.409925684Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195791.559884_260.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.559884544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.564310_261.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.564310563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.566185_262.1","section":"bucket","name":"tazcyr-27","timestamp":"2020-12-05T19:16:31.566185642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_JEiQXfZBSeXkBIrz6Wh64","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.570164_263.1","section":"bucket","name":"tazcyr-27","timestamp":"2020-12-05T19:16:31.570164527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_JEiQXfZBSeXkBIrz6Wh64","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.595904_264.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.595904056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.600782_265.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.600782574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.604311_266.1","section":"bucket","name":"tazcyr-28","timestamp":"2020-12-05T19:16:31.604311437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oc40Ilk1mlY412aky8o9il","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.611909_267.1","section":"bucket","name":"tazcyr-28","timestamp":"2020-12-05T19:16:31.611909175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oc40Ilk1mlY412aky8o9il","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.663356_268.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.663356474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.667691_269.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.667691372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.669956_270.1","section":"bucket","name":"tazcyr-29","timestamp":"2020-12-05T19:16:31.669956641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIrY6MlhiVSSiSqOOrUjlrR","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.674034_271.1","section":"bucket","name":"tazcyr-29","timestamp":"2020-12-05T19:16:31.674034946Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIrY6MlhiVSSiSqOOrUjlrR","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.698752_272.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.698752989Z","data":{"read_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.704822_273.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.704822499Z","data":{"read_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.751933_274.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.751933376Z","data":{"read_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.762446_275.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.762446815Z","data":{"read_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.823681_276.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.823681757Z","data":{"read_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.834360_277.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.834360396Z","data":{"read_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.910596_278.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:31.910596052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.916091_279.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:31.916091538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.918309_280.1","section":"bucket","name":"tazcyr-30","timestamp":"2020-12-05T19:16:31.918309043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V8JIXwA0EFqmAOaZ0sNYj58","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.922767_281.1","section":"bucket","name":"tazcyr-30","timestamp":"2020-12-05T19:16:31.922767856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V8JIXwA0EFqmAOaZ0sNYj58","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.952599_282.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:31.952599928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.956964_283.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:31.956964230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.958862_284.1","section":"bucket","name":"tazcyr-31","timestamp":"2020-12-05T19:16:31.958862465Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kBolvOrTYgZglOiG8VomhvM","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.962547_285.1","section":"bucket","name":"tazcyr-31","timestamp":"2020-12-05T19:16:31.962547883Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kBolvOrTYgZglOiG8VomhvM","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.015816_286.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.015816543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.020059_287.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.020059189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.021989_288.1","section":"bucket","name":"tazcyr-32","timestamp":"2020-12-05T19:16:32.021989230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-LCTSZbdQ2jP8bifQaJGDGi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.025960_289.1","section":"bucket","name":"tazcyr-32","timestamp":"2020-12-05T19:16:32.025960414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-LCTSZbdQ2jP8bifQaJGDGi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.049102_290.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:32.049102719Z","data":{"read_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.055546_291.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:32.055546086Z","data":{"read_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.105292_292.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:32.105292705Z","data":{"read_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.115903_293.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:32.115903395Z","data":{"read_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.173595_294.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.173595399Z","data":{"read_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.184392_295.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.184392397Z","data":{"read_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.261803_296.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.261803672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.266312_297.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.266312344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.268210_298.1","section":"bucket","name":"tazcyr-33","timestamp":"2020-12-05T19:16:32.268210820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVZLUD1vCqJ0Lt51HnkShsh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.273129_299.1","section":"bucket","name":"tazcyr-33","timestamp":"2020-12-05T19:16:32.273129813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVZLUD1vCqJ0Lt51HnkShsh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.300869_300.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:32.300869268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.306084_301.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:32.306084643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.308801_302.1","section":"bucket","name":"tazcyr-34","timestamp":"2020-12-05T19:16:32.308801370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iobgkLTDqhCewwIrJ3fJdW1","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.314464_303.1","section":"bucket","name":"tazcyr-34","timestamp":"2020-12-05T19:16:32.314464684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iobgkLTDqhCewwIrJ3fJdW1","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.370666_304.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:32.370666662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.377710_305.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:32.377710597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.379915_306.1","section":"bucket","name":"tazcyr-35","timestamp":"2020-12-05T19:16:32.379915477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lFts6vEJM81SxRXfGJh58Uh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.385846_307.1","section":"bucket","name":"tazcyr-35","timestamp":"2020-12-05T19:16:32.385846428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lFts6vEJM81SxRXfGJh58Uh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.412075_308.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.412075144Z","data":{"read_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.418632_309.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.418632526Z","data":{"read_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195793.468449_310.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:33.468449275Z","data":{"read_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":2},"status":{"status":"write"}}},{"id":"1_1607195793.480212_311.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:33.480212385Z","data":{"read_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195794.873408_312.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:34.873408400Z","data":{"read_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":2},"status":{"status":"write"}}},{"id":"1_1607195794.884443_313.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:34.884443336Z","data":{"read_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195795.745836_314.1","section":"bucket.instance","name":"tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45","timestamp":"2020-12-05T19:16:35.745836986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4ht6IvT3086xt5Cyjp-j0ZS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195795.750228_315.1","section":"bucket.instance","name":"tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45","timestamp":"2020-12-05T19:16:35.750228551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4ht6IvT3086xt5Cyjp-j0ZS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195795.752148_316.1","section":"bucket","name":"tazcyr-36","timestamp":"2020-12-05T19:16:35.752148872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oGGEcX5VQT-_WeBe3Thlum-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195795.756045_317.1","section":"bucket","name":"tazcyr-36","timestamp":"2020-12-05T19:16:35.756045544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oGGEcX5VQT-_WeBe3Thlum-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195796.395474_318.1","section":"bucket.instance","name":"tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46","timestamp":"2020-12-05T19:16:36.395474459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ynVeybecmoaDEfbRRNXnLxN","ver":1},"status":{"status":"write"}}},{"id":"1_1607195796.400062_319.1","section":"bucket.instance","name":"tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46","timestamp":"2020-12-05T19:16:36.400062414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ynVeybecmoaDEfbRRNXnLxN","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195796.402873_320.1","section":"bucket","name":"tazcyr-37","timestamp":"2020-12-05T19:16:36.402873361Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ETIrjIJ9hXuaj08WUt20_M","ver":1},"status":{"status":"write"}}},{"id":"1_1607195796.408510_321.1","section":"bucket","name":"tazcyr-37","timestamp":"2020-12-05T19:16:36.408510458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ETIrjIJ9hXuaj08WUt20_M","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195797.171740_322.1","section":"bucket.instance","name":"tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47","timestamp":"2020-12-05T19:16:37.171740685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ZQxJGge97W6bFOHGiwYws4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195797.177014_323.1","section":"bucket.instance","name":"tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47","timestamp":"2020-12-05T19:16:37.177014883Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ZQxJGge97W6bFOHGiwYws4","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195797.179628_324.1","section":"bucket","name":"tazcyr-38","timestamp":"2020-12-05T19:16:37.179628174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AamJhZR6s-WRjWe4vUMWqZr","ver":1},"status":{"status":"write"}}},{"id":"1_1607195797.184194_325.1","section":"bucket","name":"tazcyr-38","timestamp":"2020-12-05T19:16:37.184194689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AamJhZR6s-WRjWe4vUMWqZr","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195804.954889_326.1","section":"bucket.instance","name":"tazcyr-39:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48","timestamp":"2020-12-05T19:16:44.954889275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZhWn8Vk-C65I60Fnujl9kA8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195804.959965_327.1","section":"bucket.instance","name":"tazcyr-39:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48","timestamp":"2020-12-05T19:16:44.959965543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZhWn8Vk-C65I60Fnujl9kA8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195804.962048_328.1","section":"bucket","name":"tazcyr-39","timestamp":"2020-12-05T19:16:44.962048855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I5AIOg_F98j_OAJcf9wjuz-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195804.966788_329.1","section":"bucket","name":"tazcyr-39","timestamp":"2020-12-05T19:16:44.966788520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I5AIOg_F98j_OAJcf9wjuz-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.000660_330.1","section":"bucket.instance","name":"tazcyr-40:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49","timestamp":"2020-12-05T19:16:45.000660333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IN7Q23pS9W6mT4CaCf6VRbn","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.005998_331.1","section":"bucket.instance","name":"tazcyr-40:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49","timestamp":"2020-12-05T19:16:45.005998516Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IN7Q23pS9W6mT4CaCf6VRbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.008723_332.1","section":"bucket","name":"tazcyr-40","timestamp":"2020-12-05T19:16:45.008723804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VZMQhsTfCmn2-5GsJClxTBt","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.013709_333.1","section":"bucket","name":"tazcyr-40","timestamp":"2020-12-05T19:16:45.013709472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VZMQhsTfCmn2-5GsJClxTBt","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.063557_334.1","section":"bucket.instance","name":"tazcyr-41:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50","timestamp":"2020-12-05T19:16:45.063557672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KIlWf-4lWLe_zJjRcPBm7s3","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.068677_335.1","section":"bucket.instance","name":"tazcyr-41:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50","timestamp":"2020-12-05T19:16:45.068677879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KIlWf-4lWLe_zJjRcPBm7s3","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.071477_336.1","section":"bucket","name":"tazcyr-41","timestamp":"2020-12-05T19:16:45.071477845Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lXJLrvU6a2EnkXyJggSn-k4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.076974_337.1","section":"bucket","name":"tazcyr-41","timestamp":"2020-12-05T19:16:45.076974739Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lXJLrvU6a2EnkXyJggSn-k4","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:14.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:14.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:14.589 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period cb156068-23b3-4eca-8803-e19dbeb79445 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:14.628 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:14.628 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:14.933 INFO:teuthology.orchestra.run.smithi023.stdout:[{"id":"1_1607195257.607529_44.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-12-05T19:07:37.607529336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QzWxyJmSnbzgxXwE-SYm0Au","ver":1},"status":{"status":"write"}}},{"id":"1_1607195257.616573_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-12-05T19:07:37.616573496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QzWxyJmSnbzgxXwE-SYm0Au","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.004271_46.1","section":"bucket.instance","name":"tazcyr-1:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1","timestamp":"2020-12-05T19:09:40.004271326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzba4sBo5g6GI2dMc2uNhJS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.016937_47.1","section":"bucket.instance","name":"tazcyr-1:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1","timestamp":"2020-12-05T19:09:40.016937469Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzba4sBo5g6GI2dMc2uNhJS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.021417_48.1","section":"bucket","name":"tazcyr-1","timestamp":"2020-12-05T19:09:40.021417606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VyXMtA4g5xk2Opu-uU28KKS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.024544_49.1","section":"bucket","name":"tazcyr-1","timestamp":"2020-12-05T19:09:40.024544422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VyXMtA4g5xk2Opu-uU28KKS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.046042_50.1","section":"bucket.instance","name":"tazcyr-2:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2","timestamp":"2020-12-05T19:09:40.046042905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KP2FkRxiLEAiJe0MTCH9I4b","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.051202_51.1","section":"bucket.instance","name":"tazcyr-2:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2","timestamp":"2020-12-05T19:09:40.051202070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KP2FkRxiLEAiJe0MTCH9I4b","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195380.052778_52.1","section":"bucket","name":"tazcyr-2","timestamp":"2020-12-05T19:09:40.052778306Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WUyHO6QQbtRgQbl6LIL5fmT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195380.057318_53.1","section":"bucket","name":"tazcyr-2","timestamp":"2020-12-05T19:09:40.057318427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WUyHO6QQbtRgQbl6LIL5fmT","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195383.710306_54.1","section":"bucket.instance","name":"tazcyr-3:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3","timestamp":"2020-12-05T19:09:43.710306623Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mYlUpB34g7c36jCJJY7xASF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195383.720265_55.1","section":"bucket.instance","name":"tazcyr-3:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3","timestamp":"2020-12-05T19:09:43.720265984Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mYlUpB34g7c36jCJJY7xASF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195383.722055_56.1","section":"bucket","name":"tazcyr-3","timestamp":"2020-12-05T19:09:43.722055553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YJJrKeypq2XGKHx5hyNMCIx","ver":1},"status":{"status":"write"}}},{"id":"1_1607195383.731707_57.1","section":"bucket","name":"tazcyr-3","timestamp":"2020-12-05T19:09:43.731707462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YJJrKeypq2XGKHx5hyNMCIx","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.335546_60.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4","timestamp":"2020-12-05T19:09:51.335546801Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L_jTexV2QOKdHJrFnfn7kbh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.342055_61.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4","timestamp":"2020-12-05T19:09:51.342055564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L_jTexV2QOKdHJrFnfn7kbh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.344193_62.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:09:51.344193062Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jLnWUWkBmUYtUiCVFVQfp-b","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.351287_63.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:09:51.351287484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jLnWUWkBmUYtUiCVFVQfp-b","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.373162_64.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5","timestamp":"2020-12-05T19:09:51.373162509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E8T9igSLKPGjPwqoSjLEp_k","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.378587_65.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5","timestamp":"2020-12-05T19:09:51.378587248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E8T9igSLKPGjPwqoSjLEp_k","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.381243_66.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:09:51.381243284Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rz7EK0Cpc8qessr8O7Tn5iH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.388606_67.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:09:51.388606811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rz7EK0Cpc8qessr8O7Tn5iH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.437812_68.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6","timestamp":"2020-12-05T19:09:51.437812836Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZX-6V2b9MxDZwTcrSqix7n6","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.441771_69.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6","timestamp":"2020-12-05T19:09:51.441771244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZX-6V2b9MxDZwTcrSqix7n6","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195391.444005_70.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:09:51.444005575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6jyy8rxZVWt7mkAC_xVvmdF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195391.448654_71.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:09:51.448654007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6jyy8rxZVWt7mkAC_xVvmdF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.085465_72.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.7","timestamp":"2020-12-05T19:10:04.085465407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Cd-JV10QVaCV4xzrR7e8Cb","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.089157_73.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.7","timestamp":"2020-12-05T19:10:04.089157232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1Cd-JV10QVaCV4xzrR7e8Cb","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.091040_74.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.091040641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ibQYDdoT09BZvl4zzz8MPh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.094388_75.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.094388936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ibQYDdoT09BZvl4zzz8MPh","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.124511_76.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.8","timestamp":"2020-12-05T19:10:04.124511324Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9u5vcAmfWRm3ybgEgHDxyJ7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.129450_77.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.8","timestamp":"2020-12-05T19:10:04.129450232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9u5vcAmfWRm3ybgEgHDxyJ7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.132112_78.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.132112638Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QhkUsfUtoco38o0vRSTiqJm","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.137475_79.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.137475791Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QhkUsfUtoco38o0vRSTiqJm","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.164520_80.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.9","timestamp":"2020-12-05T19:10:04.164520302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8G8bhkxk9kDUQA-JQCo8M0","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.169210_81.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.9","timestamp":"2020-12-05T19:10:04.169210675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8G8bhkxk9kDUQA-JQCo8M0","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.171105_82.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.171105195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ERy_2HXt0qrGTMck-HhMHoA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.174573_83.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.174573484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ERy_2HXt0qrGTMck-HhMHoA","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.202373_84.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.10","timestamp":"2020-12-05T19:10:04.202373262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjYrzFHFxksxKI8RLAch6d7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.206658_85.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.10","timestamp":"2020-12-05T19:10:04.206658084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjYrzFHFxksxKI8RLAch6d7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.209421_86.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.209421800Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_peuGzSEo8-wqUKsHfkgtZvP","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.212892_87.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.212892151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_peuGzSEo8-wqUKsHfkgtZvP","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.271570_88.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.11","timestamp":"2020-12-05T19:10:04.271570054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GIGteFOTnsUR-BSdJ7KuGyi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.276571_89.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.11","timestamp":"2020-12-05T19:10:04.276571979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GIGteFOTnsUR-BSdJ7KuGyi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.278943_90.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.278943293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZCB1N_YTX9IpoOGN96l67YT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.286241_91.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.286241492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZCB1N_YTX9IpoOGN96l67YT","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.393793_92.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.12","timestamp":"2020-12-05T19:10:04.393793199Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wZ6ND_m9qn3va88zzyFK_lh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.398122_93.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.12","timestamp":"2020-12-05T19:10:04.398122874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wZ6ND_m9qn3va88zzyFK_lh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.400992_94.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.400992194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bvtob7IKitpBTl1g-h1fDWw","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.407785_95.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.407785961Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bvtob7IKitpBTl1g-h1fDWw","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.516948_96.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.13","timestamp":"2020-12-05T19:10:04.516948552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_S9v1ugs3p0bAN19LWmAKo","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.521889_97.1","section":"bucket.instance","name":"tazcyr-4:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.13","timestamp":"2020-12-05T19:10:04.521889148Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_S9v1ugs3p0bAN19LWmAKo","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.524060_98.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.524060551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRn-G8myGtlhg91Dk-Xg1zs","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.528641_99.1","section":"bucket","name":"tazcyr-4","timestamp":"2020-12-05T19:10:04.528641393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRn-G8myGtlhg91Dk-Xg1zs","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.572841_100.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.14","timestamp":"2020-12-05T19:10:04.572841515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VKyqbVNoVDGlDXXzerwmsua","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.577361_101.1","section":"bucket.instance","name":"tazcyr-5:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.14","timestamp":"2020-12-05T19:10:04.577361576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VKyqbVNoVDGlDXXzerwmsua","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.579161_102.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.579161032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_egt4Jp_vRt2MTLqMPIEdrsz","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.582561_103.1","section":"bucket","name":"tazcyr-5","timestamp":"2020-12-05T19:10:04.582561935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_egt4Jp_vRt2MTLqMPIEdrsz","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195404.682129_104.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.15","timestamp":"2020-12-05T19:10:04.682129773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TiQoTfba0kD_Nn8vVWkQDe8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.689318_105.1","section":"bucket.instance","name":"tazcyr-6:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.15","timestamp":"2020-12-05T19:10:04.689318302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TiQoTfba0kD_Nn8vVWkQDe8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195404.691755_106.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.691755301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJgnpoAQLxQuG357X5jW8Wb","ver":1},"status":{"status":"write"}}},{"id":"1_1607195404.697154_107.1","section":"bucket","name":"tazcyr-6","timestamp":"2020-12-05T19:10:04.697154104Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJgnpoAQLxQuG357X5jW8Wb","ver":1},"status":{"status":"abort"}}},{"id":"1_1607195417.967417_108.1","section":"bucket.instance","name":"tazcyr-7:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16","timestamp":"2020-12-05T19:10:17.967417000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PK9gOINKQY1odbbT4wek8C_","ver":1},"status":{"status":"write"}}},{"id":"1_1607195417.971501_109.1","section":"bucket.instance","name":"tazcyr-7:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16","timestamp":"2020-12-05T19:10:17.971501874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PK9gOINKQY1odbbT4wek8C_","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195417.973539_110.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:17.973539330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195417.977586_111.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:17.977586121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195417.998346_112.1","section":"bucket.instance","name":"tazcyr-8:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17","timestamp":"2020-12-05T19:10:17.998346857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i7BUcpgTpgUsW-HrIAVrF6r","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.003566_113.1","section":"bucket.instance","name":"tazcyr-8:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17","timestamp":"2020-12-05T19:10:18.003566781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i7BUcpgTpgUsW-HrIAVrF6r","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.005584_114.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:18.005584098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.009519_115.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:18.009519950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.057689_116.1","section":"bucket.instance","name":"tazcyr-9:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18","timestamp":"2020-12-05T19:10:18.057689828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BCybjCVg7kHS2ZQlWPWBlz","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.061844_117.1","section":"bucket.instance","name":"tazcyr-9:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18","timestamp":"2020-12-05T19:10:18.061844526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4BCybjCVg7kHS2ZQlWPWBlz","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195418.063969_118.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:18.063969600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195418.067947_119.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:18.067947008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195426.216787_120.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:26.216787778Z","data":{"read_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.221990_121.1","section":"bucket","name":"tazcyr-7","timestamp":"2020-12-05T19:10:26.221990508Z","data":{"read_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":1},"write_version":{"tag":"_WIMo9AcQ46p-LwEA7VVs8WH","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195426.277724_122.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:26.277724745Z","data":{"read_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.284587_123.1","section":"bucket","name":"tazcyr-8","timestamp":"2020-12-05T19:10:26.284587333Z","data":{"read_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":1},"write_version":{"tag":"_r3piTWJZP2FgQYg6LrO3Qdq","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195426.373298_124.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:26.373298757Z","data":{"read_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":2},"status":{"status":"remove"}}},{"id":"1_1607195426.378999_125.1","section":"bucket","name":"tazcyr-9","timestamp":"2020-12-05T19:10:26.378999520Z","data":{"read_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":1},"write_version":{"tag":"_R-HiLc1nFuRNaRHTt8ChFlS","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195433.462730_126.1","section":"bucket.instance","name":"tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19","timestamp":"2020-12-05T19:10:33.462730080Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_52wD5vB-WvWwe7ywIABs_eP","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.467706_127.1","section":"bucket.instance","name":"tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19","timestamp":"2020-12-05T19:10:33.467706513Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_52wD5vB-WvWwe7ywIABs_eP","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.470216_128.1","section":"bucket","name":"tazcyr-10","timestamp":"2020-12-05T19:10:33.470216766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gF0E0avjnogGfUjbNKC363v","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.476060_129.1","section":"bucket","name":"tazcyr-10","timestamp":"2020-12-05T19:10:33.476060218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gF0E0avjnogGfUjbNKC363v","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.501303_130.1","section":"bucket.instance","name":"tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20","timestamp":"2020-12-05T19:10:33.501303520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IpmsnwQB3g2CXO9fV5tmzxe","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.505700_131.1","section":"bucket.instance","name":"tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20","timestamp":"2020-12-05T19:10:33.505700636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IpmsnwQB3g2CXO9fV5tmzxe","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.510386_132.1","section":"bucket","name":"tazcyr-11","timestamp":"2020-12-05T19:10:33.510386421Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCLh27ru7alDAFNCjpuYcbo","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.515084_133.1","section":"bucket","name":"tazcyr-11","timestamp":"2020-12-05T19:10:33.515084294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCLh27ru7alDAFNCjpuYcbo","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.563793_134.1","section":"bucket.instance","name":"tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21","timestamp":"2020-12-05T19:10:33.563793962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oEVFbZgDKIx5z8F7aheS42B","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.568424_135.1","section":"bucket.instance","name":"tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21","timestamp":"2020-12-05T19:10:33.568424303Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oEVFbZgDKIx5z8F7aheS42B","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195433.570642_136.1","section":"bucket","name":"tazcyr-12","timestamp":"2020-12-05T19:10:33.570642630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7GLmb4mMYo7YxTZc8acqrd3","ver":1},"status":{"status":"write"}}},{"id":"1_1607195433.575954_137.1","section":"bucket","name":"tazcyr-12","timestamp":"2020-12-05T19:10:33.575954766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7GLmb4mMYo7YxTZc8acqrd3","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.946968_148.1","section":"bucket.instance","name":"tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22","timestamp":"2020-12-05T19:10:54.946968611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pWRHIOzF5jjrCYjz7s-bT92","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.950994_149.1","section":"bucket.instance","name":"tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22","timestamp":"2020-12-05T19:10:54.950994376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pWRHIOzF5jjrCYjz7s-bT92","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.952799_150.1","section":"bucket","name":"tazcyr-13","timestamp":"2020-12-05T19:10:54.952799584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v1sOo4IxCDRkTTtx0nqQTQR","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.957698_151.1","section":"bucket","name":"tazcyr-13","timestamp":"2020-12-05T19:10:54.957698951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v1sOo4IxCDRkTTtx0nqQTQR","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.980569_152.1","section":"bucket.instance","name":"tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23","timestamp":"2020-12-05T19:10:54.980569878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0FBdQ8v781ipPDnrIzhREbU","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.985762_153.1","section":"bucket.instance","name":"tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23","timestamp":"2020-12-05T19:10:54.985762061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0FBdQ8v781ipPDnrIzhREbU","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195454.987699_154.1","section":"bucket","name":"tazcyr-14","timestamp":"2020-12-05T19:10:54.987699674Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ff-PcqwkIaDSj0MieJiq-nF","ver":1},"status":{"status":"write"}}},{"id":"1_1607195454.991890_155.1","section":"bucket","name":"tazcyr-14","timestamp":"2020-12-05T19:10:54.991890859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ff-PcqwkIaDSj0MieJiq-nF","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195455.033052_156.1","section":"bucket.instance","name":"tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24","timestamp":"2020-12-05T19:10:55.033052994Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rSc70PDLEverIcXCOefRxJC","ver":1},"status":{"status":"write"}}},{"id":"1_1607195455.038390_157.1","section":"bucket.instance","name":"tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24","timestamp":"2020-12-05T19:10:55.038390619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rSc70PDLEverIcXCOefRxJC","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195455.040782_158.1","section":"bucket","name":"tazcyr-15","timestamp":"2020-12-05T19:10:55.040782843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dHhEsmoErlujX90WT0hZ8b-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195455.044591_159.1","section":"bucket","name":"tazcyr-15","timestamp":"2020-12-05T19:10:55.044591823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dHhEsmoErlujX90WT0hZ8b-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.156008_180.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.156008202Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.160311_181.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.160311339Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.162280_182.1","section":"bucket","name":"tazcyr-16","timestamp":"2020-12-05T19:11:44.162280128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VJ5feBj8ie73TOSNm-mOk48","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.166249_183.1","section":"bucket","name":"tazcyr-16","timestamp":"2020-12-05T19:11:44.166249418Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VJ5feBj8ie73TOSNm-mOk48","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.190323_184.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.190323520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.195436_185.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.195436227Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.197484_186.1","section":"bucket","name":"tazcyr-17","timestamp":"2020-12-05T19:11:44.197484394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A5Da6oy-XjJJ2vlamdSut1L","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.201637_187.1","section":"bucket","name":"tazcyr-17","timestamp":"2020-12-05T19:11:44.201637438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A5Da6oy-XjJJ2vlamdSut1L","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.245368_188.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.245368009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.249765_189.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.249765519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.251738_190.1","section":"bucket","name":"tazcyr-18","timestamp":"2020-12-05T19:11:44.251738627Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MLrGHAkNXQ9LQO7chnIbo9u","ver":1},"status":{"status":"write"}}},{"id":"1_1607195504.255952_191.1","section":"bucket","name":"tazcyr-18","timestamp":"2020-12-05T19:11:44.255952452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MLrGHAkNXQ9LQO7chnIbo9u","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195504.277957_192.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.277957424Z","data":{"read_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.283291_193.1","section":"bucket.instance","name":"tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25","timestamp":"2020-12-05T19:11:44.283291837Z","data":{"read_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":1},"write_version":{"tag":"_WlksOaS24iEWtjnJSF6sYds","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195504.287523_194.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.287523427Z","data":{"read_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.293539_195.1","section":"bucket.instance","name":"tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26","timestamp":"2020-12-05T19:11:44.293539973Z","data":{"read_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":1},"write_version":{"tag":"_1S5cukty90oykDk7OSGG28W","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195504.304166_196.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.304166482Z","data":{"read_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":2},"status":{"status":"write"}}},{"id":"1_1607195504.311460_197.1","section":"bucket.instance","name":"tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27","timestamp":"2020-12-05T19:11:44.311460900Z","data":{"read_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":1},"write_version":{"tag":"_nkpBASgp8bIOf0YDsxuch9v","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195609.459177_203.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.459177746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"status":{"status":"write"}}},{"id":"1_1607195609.463517_204.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.463517024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195609.465743_205.1","section":"bucket","name":"tazcyr-19","timestamp":"2020-12-05T19:13:29.465743023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sqj9c_Y5K-PaIIhhHfp9aV-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195609.469886_206.1","section":"bucket","name":"tazcyr-19","timestamp":"2020-12-05T19:13:29.469886722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sqj9c_Y5K-PaIIhhHfp9aV-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195609.476273_207.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.476273667Z","data":{"read_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":2},"status":{"status":"write"}}},{"id":"1_1607195609.483992_208.1","section":"bucket.instance","name":"tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28","timestamp":"2020-12-05T19:13:29.483992218Z","data":{"read_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":1},"write_version":{"tag":"_QVJgwrGVs2Jk9h1LlsDlLsv","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195673.656971_210.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:33.656971180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195673.662020_211.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:33.662020595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195673.664953_212.1","section":"bucket","name":"tazcyr-20","timestamp":"2020-12-05T19:14:33.664953183Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gbiHSoULZx5M93Ii9V39qaY","ver":1},"status":{"status":"write"}}},{"id":"1_1607195673.670005_213.1","section":"bucket","name":"tazcyr-20","timestamp":"2020-12-05T19:14:33.670005902Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gbiHSoULZx5M93Ii9V39qaY","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195693.006205_214.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:53.006205921Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"status":{"status":"write"}}},{"id":"1_1607195693.016790_215.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:14:53.016790934Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":1},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195713.362932_216.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:15:13.362932020Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":3},"status":{"status":"write"}}},{"id":"1_1607195713.373474_217.1","section":"bucket.instance","name":"tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29","timestamp":"2020-12-05T19:15:13.373474609Z","data":{"read_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":2},"write_version":{"tag":"_7telFHFr8MyJ7X5S569xAjA","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195727.768234_218.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.768234903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.773284_219.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.773284550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.775253_220.1","section":"bucket","name":"tazcyr-21","timestamp":"2020-12-05T19:15:27.775253122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IiMD6iqfJAhr5nl3FFCo11h","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.778878_221.1","section":"bucket","name":"tazcyr-21","timestamp":"2020-12-05T19:15:27.778878626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IiMD6iqfJAhr5nl3FFCo11h","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.805082_222.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.805082256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.810030_223.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.810030473Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.812453_224.1","section":"bucket","name":"tazcyr-22","timestamp":"2020-12-05T19:15:27.812453705Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vlaNvC_VGBzo9D9tFjShPpD","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.817658_225.1","section":"bucket","name":"tazcyr-22","timestamp":"2020-12-05T19:15:27.817658460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vlaNvC_VGBzo9D9tFjShPpD","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.872578_226.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.872578877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.877693_227.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.877693514Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.880376_228.1","section":"bucket","name":"tazcyr-23","timestamp":"2020-12-05T19:15:27.880376747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1zdqPNwasiraGjdlhZp6oQt","ver":1},"status":{"status":"write"}}},{"id":"1_1607195727.887781_229.1","section":"bucket","name":"tazcyr-23","timestamp":"2020-12-05T19:15:27.887781658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1zdqPNwasiraGjdlhZp6oQt","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195727.910906_230.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.910906788Z","data":{"read_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.915864_231.1","section":"bucket.instance","name":"tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30","timestamp":"2020-12-05T19:15:27.915864011Z","data":{"read_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":1},"write_version":{"tag":"_XP0ffIDWKGDG5I91yRqhtw6","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195727.920240_232.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.920240134Z","data":{"read_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.927455_233.1","section":"bucket.instance","name":"tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31","timestamp":"2020-12-05T19:15:27.927455161Z","data":{"read_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":1},"write_version":{"tag":"_51HNCpGqLWcno4X7rU1riMe","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195727.938873_234.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.938873524Z","data":{"read_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":2},"status":{"status":"write"}}},{"id":"1_1607195727.946934_235.1","section":"bucket.instance","name":"tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32","timestamp":"2020-12-05T19:15:27.946934371Z","data":{"read_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":1},"write_version":{"tag":"_UCYLiAihrpUugMwdctcO577","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.202717_236.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.202717397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.206641_237.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.206641166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.208613_238.1","section":"bucket","name":"tazcyr-24","timestamp":"2020-12-05T19:15:58.208613089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJ9WH9704qJNfc87y_Uym40","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.212696_239.1","section":"bucket","name":"tazcyr-24","timestamp":"2020-12-05T19:15:58.212696835Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GJ9WH9704qJNfc87y_Uym40","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.239556_240.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.239556477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.243770_241.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.243770434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.245921_242.1","section":"bucket","name":"tazcyr-25","timestamp":"2020-12-05T19:15:58.245921958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zxa0UHNz0xhSqsKfRDwukp8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.249853_243.1","section":"bucket","name":"tazcyr-25","timestamp":"2020-12-05T19:15:58.249853832Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zxa0UHNz0xhSqsKfRDwukp8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.294853_244.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.294853051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.299247_245.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.299247495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.301130_246.1","section":"bucket","name":"tazcyr-26","timestamp":"2020-12-05T19:15:58.301130640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__lGAs5PGVMg5veawjx9eKO1","ver":1},"status":{"status":"write"}}},{"id":"1_1607195758.304797_247.1","section":"bucket","name":"tazcyr-26","timestamp":"2020-12-05T19:15:58.304797057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__lGAs5PGVMg5veawjx9eKO1","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195758.324771_248.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.324771799Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.329935_249.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.329935480Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":1},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.333372_250.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.333372026Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.339373_251.1","section":"bucket.instance","name":"tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33","timestamp":"2020-12-05T19:15:58.339373051Z","data":{"read_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":2},"write_version":{"tag":"_W1PkAcZL9uUfGunhAUz2R5w","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195758.344276_252.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.344276591Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.350155_253.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.350155215Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":1},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.361818_254.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.361818610Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.369799_255.1","section":"bucket.instance","name":"tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34","timestamp":"2020-12-05T19:15:58.369799195Z","data":{"read_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":2},"write_version":{"tag":"_R03BOLDxa7QxJscv-meLjdk","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195758.382775_256.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.382775320Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"status":{"status":"write"}}},{"id":"1_1607195758.391363_257.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.391363669Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":1},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195758.402710_258.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.402710892Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":3},"status":{"status":"write"}}},{"id":"1_1607195758.409925_259.1","section":"bucket.instance","name":"tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35","timestamp":"2020-12-05T19:15:58.409925684Z","data":{"read_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":2},"write_version":{"tag":"_l9X5gdeIENxb811upGOpbW7","ver":3},"status":{"status":"complete"}}},{"id":"1_1607195791.559884_260.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.559884544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.564310_261.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.564310563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.566185_262.1","section":"bucket","name":"tazcyr-27","timestamp":"2020-12-05T19:16:31.566185642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_JEiQXfZBSeXkBIrz6Wh64","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.570164_263.1","section":"bucket","name":"tazcyr-27","timestamp":"2020-12-05T19:16:31.570164527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_S_JEiQXfZBSeXkBIrz6Wh64","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.595904_264.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.595904056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.600782_265.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.600782574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.604311_266.1","section":"bucket","name":"tazcyr-28","timestamp":"2020-12-05T19:16:31.604311437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oc40Ilk1mlY412aky8o9il","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.611909_267.1","section":"bucket","name":"tazcyr-28","timestamp":"2020-12-05T19:16:31.611909175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oc40Ilk1mlY412aky8o9il","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.663356_268.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.663356474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.667691_269.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.667691372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.669956_270.1","section":"bucket","name":"tazcyr-29","timestamp":"2020-12-05T19:16:31.669956641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIrY6MlhiVSSiSqOOrUjlrR","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.674034_271.1","section":"bucket","name":"tazcyr-29","timestamp":"2020-12-05T19:16:31.674034946Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rIrY6MlhiVSSiSqOOrUjlrR","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.698752_272.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.698752989Z","data":{"read_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.704822_273.1","section":"bucket.instance","name":"tazcyr-27:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36","timestamp":"2020-12-05T19:16:31.704822499Z","data":{"read_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":1},"write_version":{"tag":"_04JPoeqnl_2qSF_z1CgncTA","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.751933_274.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.751933376Z","data":{"read_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.762446_275.1","section":"bucket.instance","name":"tazcyr-28:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37","timestamp":"2020-12-05T19:16:31.762446815Z","data":{"read_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":1},"write_version":{"tag":"_0whb9Adz98C2yZYKqGnLEDK","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.823681_276.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.823681757Z","data":{"read_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":2},"status":{"status":"write"}}},{"id":"1_1607195791.834360_277.1","section":"bucket.instance","name":"tazcyr-29:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38","timestamp":"2020-12-05T19:16:31.834360396Z","data":{"read_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":1},"write_version":{"tag":"_nR5uG6LFa08CdOe9sp3Uun7","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195791.910596_278.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:31.910596052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.916091_279.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:31.916091538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.918309_280.1","section":"bucket","name":"tazcyr-30","timestamp":"2020-12-05T19:16:31.918309043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V8JIXwA0EFqmAOaZ0sNYj58","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.922767_281.1","section":"bucket","name":"tazcyr-30","timestamp":"2020-12-05T19:16:31.922767856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V8JIXwA0EFqmAOaZ0sNYj58","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.952599_282.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:31.952599928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.956964_283.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:31.956964230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195791.958862_284.1","section":"bucket","name":"tazcyr-31","timestamp":"2020-12-05T19:16:31.958862465Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kBolvOrTYgZglOiG8VomhvM","ver":1},"status":{"status":"write"}}},{"id":"1_1607195791.962547_285.1","section":"bucket","name":"tazcyr-31","timestamp":"2020-12-05T19:16:31.962547883Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kBolvOrTYgZglOiG8VomhvM","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.015816_286.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.015816543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.020059_287.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.020059189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.021989_288.1","section":"bucket","name":"tazcyr-32","timestamp":"2020-12-05T19:16:32.021989230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-LCTSZbdQ2jP8bifQaJGDGi","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.025960_289.1","section":"bucket","name":"tazcyr-32","timestamp":"2020-12-05T19:16:32.025960414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-LCTSZbdQ2jP8bifQaJGDGi","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.049102_290.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:32.049102719Z","data":{"read_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.055546_291.1","section":"bucket.instance","name":"tazcyr-30:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39","timestamp":"2020-12-05T19:16:32.055546086Z","data":{"read_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":1},"write_version":{"tag":"_H4Pb2jzpES5OGXBnyfox_6M","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.105292_292.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:32.105292705Z","data":{"read_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.115903_293.1","section":"bucket.instance","name":"tazcyr-31:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40","timestamp":"2020-12-05T19:16:32.115903395Z","data":{"read_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":1},"write_version":{"tag":"_1NVtPkE1EUvhdVknhp5ngVi","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.173595_294.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.173595399Z","data":{"read_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.184392_295.1","section":"bucket.instance","name":"tazcyr-32:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41","timestamp":"2020-12-05T19:16:32.184392397Z","data":{"read_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":1},"write_version":{"tag":"_ZzaZ7s8QgrCbqK1nc6bCgDm","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195792.261803_296.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.261803672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.266312_297.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.266312344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.268210_298.1","section":"bucket","name":"tazcyr-33","timestamp":"2020-12-05T19:16:32.268210820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVZLUD1vCqJ0Lt51HnkShsh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.273129_299.1","section":"bucket","name":"tazcyr-33","timestamp":"2020-12-05T19:16:32.273129813Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QVZLUD1vCqJ0Lt51HnkShsh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.300869_300.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:32.300869268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.306084_301.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:32.306084643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.308801_302.1","section":"bucket","name":"tazcyr-34","timestamp":"2020-12-05T19:16:32.308801370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iobgkLTDqhCewwIrJ3fJdW1","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.314464_303.1","section":"bucket","name":"tazcyr-34","timestamp":"2020-12-05T19:16:32.314464684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iobgkLTDqhCewwIrJ3fJdW1","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.370666_304.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:32.370666662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.377710_305.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:32.377710597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.379915_306.1","section":"bucket","name":"tazcyr-35","timestamp":"2020-12-05T19:16:32.379915477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lFts6vEJM81SxRXfGJh58Uh","ver":1},"status":{"status":"write"}}},{"id":"1_1607195792.385846_307.1","section":"bucket","name":"tazcyr-35","timestamp":"2020-12-05T19:16:32.385846428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lFts6vEJM81SxRXfGJh58Uh","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195792.412075_308.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.412075144Z","data":{"read_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":2},"status":{"status":"write"}}},{"id":"1_1607195792.418632_309.1","section":"bucket.instance","name":"tazcyr-33:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42","timestamp":"2020-12-05T19:16:32.418632526Z","data":{"read_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":1},"write_version":{"tag":"_SxB775p005458nQ4m7XNvgd","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195793.468449_310.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:33.468449275Z","data":{"read_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":2},"status":{"status":"write"}}},{"id":"1_1607195793.480212_311.1","section":"bucket.instance","name":"tazcyr-34:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43","timestamp":"2020-12-05T19:16:33.480212385Z","data":{"read_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":1},"write_version":{"tag":"_j1xgdAvK85e-GI95LvQrV5Y","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195794.873408_312.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:34.873408400Z","data":{"read_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":2},"status":{"status":"write"}}},{"id":"1_1607195794.884443_313.1","section":"bucket.instance","name":"tazcyr-35:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44","timestamp":"2020-12-05T19:16:34.884443336Z","data":{"read_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":1},"write_version":{"tag":"__yZ_aC46k3Y2wR9fSZ3iHqH","ver":2},"status":{"status":"complete"}}},{"id":"1_1607195795.745836_314.1","section":"bucket.instance","name":"tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45","timestamp":"2020-12-05T19:16:35.745836986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4ht6IvT3086xt5Cyjp-j0ZS","ver":1},"status":{"status":"write"}}},{"id":"1_1607195795.750228_315.1","section":"bucket.instance","name":"tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45","timestamp":"2020-12-05T19:16:35.750228551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4ht6IvT3086xt5Cyjp-j0ZS","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195795.752148_316.1","section":"bucket","name":"tazcyr-36","timestamp":"2020-12-05T19:16:35.752148872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oGGEcX5VQT-_WeBe3Thlum-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195795.756045_317.1","section":"bucket","name":"tazcyr-36","timestamp":"2020-12-05T19:16:35.756045544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oGGEcX5VQT-_WeBe3Thlum-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195796.395474_318.1","section":"bucket.instance","name":"tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46","timestamp":"2020-12-05T19:16:36.395474459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ynVeybecmoaDEfbRRNXnLxN","ver":1},"status":{"status":"write"}}},{"id":"1_1607195796.400062_319.1","section":"bucket.instance","name":"tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46","timestamp":"2020-12-05T19:16:36.400062414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ynVeybecmoaDEfbRRNXnLxN","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195796.402873_320.1","section":"bucket","name":"tazcyr-37","timestamp":"2020-12-05T19:16:36.402873361Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ETIrjIJ9hXuaj08WUt20_M","ver":1},"status":{"status":"write"}}},{"id":"1_1607195796.408510_321.1","section":"bucket","name":"tazcyr-37","timestamp":"2020-12-05T19:16:36.408510458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3ETIrjIJ9hXuaj08WUt20_M","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195797.171740_322.1","section":"bucket.instance","name":"tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47","timestamp":"2020-12-05T19:16:37.171740685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ZQxJGge97W6bFOHGiwYws4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195797.177014_323.1","section":"bucket.instance","name":"tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47","timestamp":"2020-12-05T19:16:37.177014883Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2ZQxJGge97W6bFOHGiwYws4","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195797.179628_324.1","section":"bucket","name":"tazcyr-38","timestamp":"2020-12-05T19:16:37.179628174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AamJhZR6s-WRjWe4vUMWqZr","ver":1},"status":{"status":"write"}}},{"id":"1_1607195797.184194_325.1","section":"bucket","name":"tazcyr-38","timestamp":"2020-12-05T19:16:37.184194689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AamJhZR6s-WRjWe4vUMWqZr","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195804.954889_326.1","section":"bucket.instance","name":"tazcyr-39:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48","timestamp":"2020-12-05T19:16:44.954889275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZhWn8Vk-C65I60Fnujl9kA8","ver":1},"status":{"status":"write"}}},{"id":"1_1607195804.959965_327.1","section":"bucket.instance","name":"tazcyr-39:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48","timestamp":"2020-12-05T19:16:44.959965543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZhWn8Vk-C65I60Fnujl9kA8","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195804.962048_328.1","section":"bucket","name":"tazcyr-39","timestamp":"2020-12-05T19:16:44.962048855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I5AIOg_F98j_OAJcf9wjuz-","ver":1},"status":{"status":"write"}}},{"id":"1_1607195804.966788_329.1","section":"bucket","name":"tazcyr-39","timestamp":"2020-12-05T19:16:44.966788520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I5AIOg_F98j_OAJcf9wjuz-","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.000660_330.1","section":"bucket.instance","name":"tazcyr-40:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49","timestamp":"2020-12-05T19:16:45.000660333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IN7Q23pS9W6mT4CaCf6VRbn","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.005998_331.1","section":"bucket.instance","name":"tazcyr-40:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49","timestamp":"2020-12-05T19:16:45.005998516Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IN7Q23pS9W6mT4CaCf6VRbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.008723_332.1","section":"bucket","name":"tazcyr-40","timestamp":"2020-12-05T19:16:45.008723804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VZMQhsTfCmn2-5GsJClxTBt","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.013709_333.1","section":"bucket","name":"tazcyr-40","timestamp":"2020-12-05T19:16:45.013709472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VZMQhsTfCmn2-5GsJClxTBt","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.063557_334.1","section":"bucket.instance","name":"tazcyr-41:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50","timestamp":"2020-12-05T19:16:45.063557672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KIlWf-4lWLe_zJjRcPBm7s3","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.068677_335.1","section":"bucket.instance","name":"tazcyr-41:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50","timestamp":"2020-12-05T19:16:45.068677879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KIlWf-4lWLe_zJjRcPBm7s3","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195805.071477_336.1","section":"bucket","name":"tazcyr-41","timestamp":"2020-12-05T19:16:45.071477845Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lXJLrvU6a2EnkXyJggSn-k4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195805.076974_337.1","section":"bucket","name":"tazcyr-41","timestamp":"2020-12-05T19:16:45.076974739Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lXJLrvU6a2EnkXyJggSn-k4","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:14.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:14.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:14.937 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 88dc4e82-8825-4cad-908f-99aa84fb6061 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:14.977 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:14.977 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:15.225 INFO:teuthology.orchestra.run.smithi185.stdout:[{"id":"1_1607195878.185818_408.1","section":"bucket.instance","name":"tazcyr-42:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1","timestamp":"2020-12-05T19:17:58.185818598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nIVCNGa7aC2tc_r6Qm91c5I","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.196629_409.1","section":"bucket.instance","name":"tazcyr-42:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1","timestamp":"2020-12-05T19:17:58.196629197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nIVCNGa7aC2tc_r6Qm91c5I","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.198824_410.1","section":"bucket","name":"tazcyr-42","timestamp":"2020-12-05T19:17:58.198824127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O8yM7s64PBiYlsZmv1u9EJY","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.208077_411.1","section":"bucket","name":"tazcyr-42","timestamp":"2020-12-05T19:17:58.208077926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O8yM7s64PBiYlsZmv1u9EJY","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.265331_412.1","section":"bucket.instance","name":"tazcyr-43:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2","timestamp":"2020-12-05T19:17:58.265331416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H8dpJhwLq9TgTdH57WXmkU_","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.269811_413.1","section":"bucket.instance","name":"tazcyr-43:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2","timestamp":"2020-12-05T19:17:58.269811664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H8dpJhwLq9TgTdH57WXmkU_","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.271736_414.1","section":"bucket","name":"tazcyr-43","timestamp":"2020-12-05T19:17:58.271736360Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bM8JYjfuxfSUXFiFQgPmKiT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.275808_415.1","section":"bucket","name":"tazcyr-43","timestamp":"2020-12-05T19:17:58.275808346Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bM8JYjfuxfSUXFiFQgPmKiT","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:15.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:15.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:15.227 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 88dc4e82-8825-4cad-908f-99aa84fb6061 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:15.266 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:15.267 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:15.522 INFO:teuthology.orchestra.run.smithi023.stdout:[{"id":"1_1607195878.185818_408.1","section":"bucket.instance","name":"tazcyr-42:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1","timestamp":"2020-12-05T19:17:58.185818598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nIVCNGa7aC2tc_r6Qm91c5I","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.196629_409.1","section":"bucket.instance","name":"tazcyr-42:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1","timestamp":"2020-12-05T19:17:58.196629197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nIVCNGa7aC2tc_r6Qm91c5I","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.198824_410.1","section":"bucket","name":"tazcyr-42","timestamp":"2020-12-05T19:17:58.198824127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O8yM7s64PBiYlsZmv1u9EJY","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.208077_411.1","section":"bucket","name":"tazcyr-42","timestamp":"2020-12-05T19:17:58.208077926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O8yM7s64PBiYlsZmv1u9EJY","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.265331_412.1","section":"bucket.instance","name":"tazcyr-43:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2","timestamp":"2020-12-05T19:17:58.265331416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H8dpJhwLq9TgTdH57WXmkU_","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.269811_413.1","section":"bucket.instance","name":"tazcyr-43:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2","timestamp":"2020-12-05T19:17:58.269811664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H8dpJhwLq9TgTdH57WXmkU_","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.271736_414.1","section":"bucket","name":"tazcyr-43","timestamp":"2020-12-05T19:17:58.271736360Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bM8JYjfuxfSUXFiFQgPmKiT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.275808_415.1","section":"bucket","name":"tazcyr-43","timestamp":"2020-12-05T19:17:58.275808346Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bM8JYjfuxfSUXFiFQgPmKiT","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:15.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:15.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:15.524 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 88dc4e82-8825-4cad-908f-99aa84fb6061 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:15.562 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:15.562 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:15.811 INFO:teuthology.orchestra.run.smithi185.stdout:[{"id":"1_1607195878.185818_408.1","section":"bucket.instance","name":"tazcyr-42:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1","timestamp":"2020-12-05T19:17:58.185818598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nIVCNGa7aC2tc_r6Qm91c5I","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.196629_409.1","section":"bucket.instance","name":"tazcyr-42:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1","timestamp":"2020-12-05T19:17:58.196629197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nIVCNGa7aC2tc_r6Qm91c5I","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.198824_410.1","section":"bucket","name":"tazcyr-42","timestamp":"2020-12-05T19:17:58.198824127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O8yM7s64PBiYlsZmv1u9EJY","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.208077_411.1","section":"bucket","name":"tazcyr-42","timestamp":"2020-12-05T19:17:58.208077926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O8yM7s64PBiYlsZmv1u9EJY","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.265331_412.1","section":"bucket.instance","name":"tazcyr-43:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2","timestamp":"2020-12-05T19:17:58.265331416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H8dpJhwLq9TgTdH57WXmkU_","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.269811_413.1","section":"bucket.instance","name":"tazcyr-43:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2","timestamp":"2020-12-05T19:17:58.269811664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H8dpJhwLq9TgTdH57WXmkU_","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.271736_414.1","section":"bucket","name":"tazcyr-43","timestamp":"2020-12-05T19:17:58.271736360Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bM8JYjfuxfSUXFiFQgPmKiT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.275808_415.1","section":"bucket","name":"tazcyr-43","timestamp":"2020-12-05T19:17:58.275808346Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bM8JYjfuxfSUXFiFQgPmKiT","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:15.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:15.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:15.813 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 88dc4e82-8825-4cad-908f-99aa84fb6061 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:15.854 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:15.854 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:16.106 INFO:teuthology.orchestra.run.smithi023.stdout:[{"id":"1_1607195878.185818_408.1","section":"bucket.instance","name":"tazcyr-42:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1","timestamp":"2020-12-05T19:17:58.185818598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nIVCNGa7aC2tc_r6Qm91c5I","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.196629_409.1","section":"bucket.instance","name":"tazcyr-42:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1","timestamp":"2020-12-05T19:17:58.196629197Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nIVCNGa7aC2tc_r6Qm91c5I","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.198824_410.1","section":"bucket","name":"tazcyr-42","timestamp":"2020-12-05T19:17:58.198824127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O8yM7s64PBiYlsZmv1u9EJY","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.208077_411.1","section":"bucket","name":"tazcyr-42","timestamp":"2020-12-05T19:17:58.208077926Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_O8yM7s64PBiYlsZmv1u9EJY","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.265331_412.1","section":"bucket.instance","name":"tazcyr-43:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2","timestamp":"2020-12-05T19:17:58.265331416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H8dpJhwLq9TgTdH57WXmkU_","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.269811_413.1","section":"bucket.instance","name":"tazcyr-43:3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2","timestamp":"2020-12-05T19:17:58.269811664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H8dpJhwLq9TgTdH57WXmkU_","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195878.271736_414.1","section":"bucket","name":"tazcyr-43","timestamp":"2020-12-05T19:17:58.271736360Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bM8JYjfuxfSUXFiFQgPmKiT","ver":1},"status":{"status":"write"}}},{"id":"1_1607195878.275808_415.1","section":"bucket","name":"tazcyr-43","timestamp":"2020-12-05T19:17:58.275808346Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bM8JYjfuxfSUXFiFQgPmKiT","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:16.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:16.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:16.109 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 99a897cd-76db-49ef-bcce-15cfc0f426ad --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:16.147 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:16.148 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:16.398 INFO:teuthology.orchestra.run.smithi185.stdout:[{"id":"1_1607195944.763674_247.1","section":"bucket.instance","name":"tazcyr-44:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1","timestamp":"2020-12-05T19:19:04.763674295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_reMxdshkldUnWD6O0ck1tNy","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.768595_248.1","section":"bucket.instance","name":"tazcyr-44:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1","timestamp":"2020-12-05T19:19:04.768595904Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_reMxdshkldUnWD6O0ck1tNy","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.770856_249.1","section":"bucket","name":"tazcyr-44","timestamp":"2020-12-05T19:19:04.770856328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KGqm9Po7sSGI5R2mCa9yxu0","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.775081_250.1","section":"bucket","name":"tazcyr-44","timestamp":"2020-12-05T19:19:04.775081258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KGqm9Po7sSGI5R2mCa9yxu0","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.814863_251.1","section":"bucket.instance","name":"tazcyr-45:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2","timestamp":"2020-12-05T19:19:04.814863694Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jeGGZpu2tbwRjMy5vByAGFd","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.819738_252.1","section":"bucket.instance","name":"tazcyr-45:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2","timestamp":"2020-12-05T19:19:04.819738927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jeGGZpu2tbwRjMy5vByAGFd","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.821778_253.1","section":"bucket","name":"tazcyr-45","timestamp":"2020-12-05T19:19:04.821778574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UNOjFD0MP4lLLcoj8Z9yYz4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.825629_254.1","section":"bucket","name":"tazcyr-45","timestamp":"2020-12-05T19:19:04.825629929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UNOjFD0MP4lLLcoj8Z9yYz4","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:16.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:16.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:16.400 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 99a897cd-76db-49ef-bcce-15cfc0f426ad --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:16.703 INFO:teuthology.orchestra.run.smithi023.stdout:[{"id":"1_1607195944.763674_247.1","section":"bucket.instance","name":"tazcyr-44:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1","timestamp":"2020-12-05T19:19:04.763674295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_reMxdshkldUnWD6O0ck1tNy","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.768595_248.1","section":"bucket.instance","name":"tazcyr-44:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1","timestamp":"2020-12-05T19:19:04.768595904Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_reMxdshkldUnWD6O0ck1tNy","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.770856_249.1","section":"bucket","name":"tazcyr-44","timestamp":"2020-12-05T19:19:04.770856328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KGqm9Po7sSGI5R2mCa9yxu0","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.775081_250.1","section":"bucket","name":"tazcyr-44","timestamp":"2020-12-05T19:19:04.775081258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KGqm9Po7sSGI5R2mCa9yxu0","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.814863_251.1","section":"bucket.instance","name":"tazcyr-45:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2","timestamp":"2020-12-05T19:19:04.814863694Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jeGGZpu2tbwRjMy5vByAGFd","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.819738_252.1","section":"bucket.instance","name":"tazcyr-45:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2","timestamp":"2020-12-05T19:19:04.819738927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jeGGZpu2tbwRjMy5vByAGFd","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.821778_253.1","section":"bucket","name":"tazcyr-45","timestamp":"2020-12-05T19:19:04.821778574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UNOjFD0MP4lLLcoj8Z9yYz4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.825629_254.1","section":"bucket","name":"tazcyr-45","timestamp":"2020-12-05T19:19:04.825629929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UNOjFD0MP4lLLcoj8Z9yYz4","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:16.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:16.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:16.704 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 99a897cd-76db-49ef-bcce-15cfc0f426ad --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:16.743 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:16.743 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:16.996 INFO:teuthology.orchestra.run.smithi185.stdout:[{"id":"1_1607195944.763674_247.1","section":"bucket.instance","name":"tazcyr-44:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1","timestamp":"2020-12-05T19:19:04.763674295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_reMxdshkldUnWD6O0ck1tNy","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.768595_248.1","section":"bucket.instance","name":"tazcyr-44:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1","timestamp":"2020-12-05T19:19:04.768595904Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_reMxdshkldUnWD6O0ck1tNy","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.770856_249.1","section":"bucket","name":"tazcyr-44","timestamp":"2020-12-05T19:19:04.770856328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KGqm9Po7sSGI5R2mCa9yxu0","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.775081_250.1","section":"bucket","name":"tazcyr-44","timestamp":"2020-12-05T19:19:04.775081258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KGqm9Po7sSGI5R2mCa9yxu0","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.814863_251.1","section":"bucket.instance","name":"tazcyr-45:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2","timestamp":"2020-12-05T19:19:04.814863694Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jeGGZpu2tbwRjMy5vByAGFd","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.819738_252.1","section":"bucket.instance","name":"tazcyr-45:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2","timestamp":"2020-12-05T19:19:04.819738927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jeGGZpu2tbwRjMy5vByAGFd","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.821778_253.1","section":"bucket","name":"tazcyr-45","timestamp":"2020-12-05T19:19:04.821778574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UNOjFD0MP4lLLcoj8Z9yYz4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.825629_254.1","section":"bucket","name":"tazcyr-45","timestamp":"2020-12-05T19:19:04.825629929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UNOjFD0MP4lLLcoj8Z9yYz4","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:16.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:16.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:16.998 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 99a897cd-76db-49ef-bcce-15cfc0f426ad --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:17.039 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:17.039 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:17.300 INFO:teuthology.orchestra.run.smithi023.stdout:[{"id":"1_1607195944.763674_247.1","section":"bucket.instance","name":"tazcyr-44:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1","timestamp":"2020-12-05T19:19:04.763674295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_reMxdshkldUnWD6O0ck1tNy","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.768595_248.1","section":"bucket.instance","name":"tazcyr-44:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1","timestamp":"2020-12-05T19:19:04.768595904Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_reMxdshkldUnWD6O0ck1tNy","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.770856_249.1","section":"bucket","name":"tazcyr-44","timestamp":"2020-12-05T19:19:04.770856328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KGqm9Po7sSGI5R2mCa9yxu0","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.775081_250.1","section":"bucket","name":"tazcyr-44","timestamp":"2020-12-05T19:19:04.775081258Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KGqm9Po7sSGI5R2mCa9yxu0","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.814863_251.1","section":"bucket.instance","name":"tazcyr-45:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2","timestamp":"2020-12-05T19:19:04.814863694Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jeGGZpu2tbwRjMy5vByAGFd","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.819738_252.1","section":"bucket.instance","name":"tazcyr-45:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2","timestamp":"2020-12-05T19:19:04.819738927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jeGGZpu2tbwRjMy5vByAGFd","ver":1},"status":{"status":"complete"}}},{"id":"1_1607195944.821778_253.1","section":"bucket","name":"tazcyr-45","timestamp":"2020-12-05T19:19:04.821778574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UNOjFD0MP4lLLcoj8Z9yYz4","ver":1},"status":{"status":"write"}}},{"id":"1_1607195944.825629_254.1","section":"bucket","name":"tazcyr-45","timestamp":"2020-12-05T19:19:04.825629929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UNOjFD0MP4lLLcoj8Z9yYz4","ver":1},"status":{"status":"complete"}}}] 2020-12-05T19:19:17.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:17.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:17.301 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:19:17.336 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:17.336 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:17.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:17.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:17.615 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:19:17.652 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:17.652 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:18.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:18.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:18.032 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:19:18.067 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:18.067 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:18.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:18.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:18.318 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:19:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:18.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:18.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:18.712 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period cb156068-23b3-4eca-8803-e19dbeb79445 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:18.751 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:18.751 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:19.000 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:19:19.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:19.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:19.001 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period cb156068-23b3-4eca-8803-e19dbeb79445 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:19.041 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:19.041 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:19.296 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2020-12-05T19:19:19.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:19.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:19.297 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period cb156068-23b3-4eca-8803-e19dbeb79445 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:19.336 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:19.336 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:19.584 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:19:19.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:19.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'cb156068-23b3-4eca-8803-e19dbeb79445', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:19.586 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period cb156068-23b3-4eca-8803-e19dbeb79445 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:19.621 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:19.622 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:19.866 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2020-12-05T19:19:19.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:19.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:19.868 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 88dc4e82-8825-4cad-908f-99aa84fb6061 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:19.904 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:19.904 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:20.146 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:19:20.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:20.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:20.148 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 88dc4e82-8825-4cad-908f-99aa84fb6061 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:20.185 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:20.186 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:20.440 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2020-12-05T19:19:20.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:20.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:20.442 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 88dc4e82-8825-4cad-908f-99aa84fb6061 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:20.479 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:20.479 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:20.734 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:19:20.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:20.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '88dc4e82-8825-4cad-908f-99aa84fb6061', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:20.736 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 88dc4e82-8825-4cad-908f-99aa84fb6061 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:21.028 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2020-12-05T19:19:21.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:21.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:21.029 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 99a897cd-76db-49ef-bcce-15cfc0f426ad --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:21.067 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:21.068 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:21.326 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:19:21.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:21.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:21.327 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 99a897cd-76db-49ef-bcce-15cfc0f426ad --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:21.634 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2020-12-05T19:19:21.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:21.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:21.636 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 99a897cd-76db-49ef-bcce-15cfc0f426ad --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:21.675 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:21.676 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:21.932 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:19:21.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:21.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '99a897cd-76db-49ef-bcce-15cfc0f426ad', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:21.933 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 99a897cd-76db-49ef-bcce-15cfc0f426ad --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:21.975 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:21.975 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:22.232 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2020-12-05T19:19:22.233 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-12-05T19:19:22.234 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-46 2020-12-05T19:19:22.858 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-47 2020-12-05T19:19:23.222 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-48 2020-12-05T19:19:23.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:23.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:23.580 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:23.617 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:23.618 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:23.867 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195963.439545_133.1","last_update":"2020-12-05T19:19:23.439545Z"},{"marker":"1_1607195624.108585_165.1","last_update":"2020-12-05T19:13:44.108585Z"},{"marker":"1_1607195806.116793_231.1","last_update":"2020-12-05T19:16:46.116793Z"},{"marker":"1_1607195580.099522_101.3","last_update":"2020-12-05T19:13:00.099522Z"}] 2020-12-05T19:19:23.867 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:19:23.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:23.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:23.869 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:23.925 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:23.925 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:24.162 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195778.129247_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:18.129247Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.108585_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.108585Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:19:24.163 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607195963.439545_133.1 target=1_1607195778.129247_98.3 2020-12-05T19:19:29.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:29.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:29.166 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:29.209 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:29.210 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:29.465 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.108585_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.108585Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:19:29.466 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:19:29.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:29.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:29.468 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:29.507 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:29.507 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:29.758 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195965.209376_136.1","last_update":"2020-12-05T19:19:25.209376Z"},{"marker":"1_1607195624.108585_165.1","last_update":"2020-12-05T19:13:44.108585Z"},{"marker":"1_1607195806.116793_231.1","last_update":"2020-12-05T19:16:46.116793Z"},{"marker":"1_1607195580.099522_101.3","last_update":"2020-12-05T19:13:00.099522Z"}] 2020-12-05T19:19:29.758 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:19:29.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:29.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:29.760 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:29.839 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:29.839 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:30.049 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10689636149165243743},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.108585_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.108585Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:19:30.050 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:19:30.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:30.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:30.052 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:30.127 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:30.128 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:30.378 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195965.209376_136.1","last_update":"2020-12-05T19:19:25.209376Z"},{"marker":"1_1607195624.108585_165.1","last_update":"2020-12-05T19:13:44.108585Z"},{"marker":"1_1607195806.116793_231.1","last_update":"2020-12-05T19:16:46.116793Z"},{"marker":"1_1607195580.099522_101.3","last_update":"2020-12-05T19:13:00.099522Z"}] 2020-12-05T19:19:30.379 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:19:30.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:30.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:30.381 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:30.422 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:30.423 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:30.701 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.108585_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.108585Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:19:30.702 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:19:30.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:30.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:30.703 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:30.782 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:30.782 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:31.023 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607195964.755529_137.1","last_update":"2020-12-05T19:19:24.755529Z"},{"marker":"1_1607195625.242812_162.1","last_update":"2020-12-05T19:13:45.242812Z"},{"marker":"1_1607195806.040983_226.1","last_update":"2020-12-05T19:16:46.040983Z"},{"marker":"1_1607195581.234035_126.3","last_update":"2020-12-05T19:13:01.234035Z"}] 2020-12-05T19:19:31.023 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:19:31.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:31.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:31.025 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:31.070 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:31.070 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:31.315 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9760760649708652305},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195964.755529_137.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:24.755529Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195625.242812_162.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:45.242812Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.040983_226.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.040983Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195581.234035_126.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:01.234035Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:19:31.315 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:19:31.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:31.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:31.317 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:31.609 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607195964.755529_137.1","last_update":"2020-12-05T19:19:24.755529Z"},{"marker":"1_1607195625.242812_162.1","last_update":"2020-12-05T19:13:45.242812Z"},{"marker":"1_1607195806.040983_226.1","last_update":"2020-12-05T19:16:46.040983Z"},{"marker":"1_1607195581.234035_126.3","last_update":"2020-12-05T19:13:01.234035Z"}] 2020-12-05T19:19:31.610 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:19:31.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:31.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:31.611 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:31.653 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:31.654 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:31.893 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3761623777010785199},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195964.755529_137.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:24.755529Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195625.242812_162.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:45.242812Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.040983_226.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.040983Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195581.234035_126.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:01.234035Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:19:31.893 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:19:31.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:31.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:31.895 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:31.970 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:31.970 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:32.198 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195964.751155_111.1","last_update":"2020-12-05T19:19:24.751155Z"},{"marker":"1_1607195624.110351_128.1","last_update":"2020-12-05T19:13:44.110351Z"},{"marker":"1_1607195805.921402_148.1","last_update":"2020-12-05T19:16:45.921402Z"},{"marker":"1_1607195580.100307_98.3","last_update":"2020-12-05T19:13:00.100307Z"}] 2020-12-05T19:19:32.198 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:19:32.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:32.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:32.200 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:32.279 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:32.279 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:32.524 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4132799784717954996},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195964.751155_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:24.751155Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.110351_128.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.110351Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195805.921402_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:45.921402Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.100307_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.100307Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:19:32.525 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:19:32.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:32.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:32.526 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:32.604 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:32.605 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:32.838 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607195964.751155_111.1","last_update":"2020-12-05T19:19:24.751155Z"},{"marker":"1_1607195624.110351_128.1","last_update":"2020-12-05T19:13:44.110351Z"},{"marker":"1_1607195805.921402_148.1","last_update":"2020-12-05T19:16:45.921402Z"},{"marker":"1_1607195580.100307_98.3","last_update":"2020-12-05T19:13:00.100307Z"}] 2020-12-05T19:19:32.839 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:19:32.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:32.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:32.840 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:32.882 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:32.882 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:33.141 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6833204601607133234},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195964.751155_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:24.751155Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607195624.110351_128.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:44.110351Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195805.921402_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:45.921402Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.100307_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.100307Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:19:33.142 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:19:33.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:33.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:33.144 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:19:33.181 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:33.182 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:33.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:33.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:33.437 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:33.515 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:33.515 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:33.765 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:19:33.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:33.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:33.767 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:19:33.801 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:33.802 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:34.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:34.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:34.057 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:34.133 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:34.133 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:34.388 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2020-12-05T19:19:34.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:34.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:34.389 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:19:34.425 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:34.426 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:34.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:34.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:19:34.682 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:19:34.757 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:34.757 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:35.001 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:19:35.002 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-12-05T19:19:35.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:35.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:35.004 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:19:35.045 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:35.046 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:19:35.068+0000 7fd3d2fdf080 0 failed reading obj info from .rgw.root:zone_info.e622b2e3-1a91-4dc1-b31e-43ede659e70f: (2) No such file or directory 2020-12-05T19:19:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:19:35.068+0000 7fd3d2fdf080 0 WARNING: could not read zone params for zone id=e622b2e3-1a91-4dc1-b31e-43ede659e70f name=test-zone1 2020-12-05T19:19:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:19:35.072+0000 7fd3d2fdf080 0 failed reading obj info from .rgw.root:zone_info.ecced906-ad37-41c3-ad42-864c1e77463f: (2) No such file or directory 2020-12-05T19:19:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:19:35.072+0000 7fd3d2fdf080 0 WARNING: could not read zone params for zone id=ecced906-ad37-41c3-ad42-864c1e77463f name=test-zone3 2020-12-05T19:19:35.101 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:19:35.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:35.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:19:35.103 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:19:35.183 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:19:35.183 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:19:35.434 INFO:teuthology.orchestra.run.smithi023.stderr:Sending period to new master zone e622b2e3-1a91-4dc1-b31e-43ede659e70f 2020-12-05T19:19:36.550 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","epoch":2,"predecessor_uuid":"88dc4e82-8825-4cad-908f-99aa84fb6061","sync_status":["","1_1607195878.275808_415.1","",""],"period_map":{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":4} 2020-12-05T19:19:36.551 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-12-05T19:20:36.551 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-49 2020-12-05T19:20:36.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:36.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:36.648 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:20:36.687 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:20:36.687 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:20:36.916 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:20:36.944 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196036.591269_315.1","last_update":"2020-12-05T19:20:36.591269Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:20:36.944 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:20:36.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:36.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:36.946 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:20:36.986 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:20:36.987 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:20:37.242 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:20:37.243 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:20:42.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:42.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:42.245 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:20:42.288 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:20:42.288 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:20:42.544 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:20:42.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:20:47.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:47.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:47.547 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:20:47.588 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:20:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:20:47.855 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:20:47.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:20:52.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:52.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:52.858 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:20:52.900 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:20:52.900 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:20:53.158 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:20:53.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:20:58.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:58.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:20:58.162 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:20:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:20:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:20:58.451 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:20:58.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:21:03.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:03.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:03.455 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:03.497 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:03.497 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:03.752 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:03.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:21:07.733 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:21:07.726+0000 7f7f38fe9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:21:08.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:08.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:08.755 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:08.795 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:08.796 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:09.036 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196036.591269_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:20:36.591269Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:09.036 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:21:09.037 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:21:09.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:09.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:09.038 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:09.077 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:09.078 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:09.325 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196036.591269_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:20:36.591269Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:09.326 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:21:09.327 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:21:09.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:09.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:09.328 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:09.367 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:09.651 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:09.652 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:21:14.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:14.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:14.654 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:14.696 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:14.697 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:14.963 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:14.964 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:21:19.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:19.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:19.967 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:20.008 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:20.009 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:20.289 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:20.290 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:21:25.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:25.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:25.293 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:25.334 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:25.335 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:25.609 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:25.609 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:21:30.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:30.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:30.612 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:30.652 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:30.652 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:30.928 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:30.929 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:21:35.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:35.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:35.931 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:35.972 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:36.249 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607195963.406064_284.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:23.406064Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:36.250 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1607196036.591269_315.1 target=1_1607195963.406064_284.1 2020-12-05T19:21:37.739 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:21:37.734+0000 7f6ecc7f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:21:41.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:41.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:21:41.252 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:21:41.292 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:41.292 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:41.566 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196036.591269_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:20:36.591269Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:21:41.567 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:21:41.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=e622b2e3-1a91-4dc1-b31e-43ede659e70f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:21:41.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=e622b2e3-1a91-4dc1-b31e-43ede659e70f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:21:41.580 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=e622b2e3-1a91-4dc1-b31e-43ede659e70f --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:21:41.651 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:41.652 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:21:41.678+0000 7f9593747080 0 failed reading obj info from .rgw.root:zone_info.e622b2e3-1a91-4dc1-b31e-43ede659e70f: (2) No such file or directory 2020-12-05T19:21:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:21:41.678+0000 7f9593747080 0 WARNING: could not read zone params for zone id=e622b2e3-1a91-4dc1-b31e-43ede659e70f name=test-zone1 2020-12-05T19:21:41.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:21:41.678+0000 7f9593747080 0 failed reading obj info from .rgw.root:zone_info.ecced906-ad37-41c3-ad42-864c1e77463f: (2) No such file or directory 2020-12-05T19:21:41.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:21:41.678+0000 7f9593747080 0 WARNING: could not read zone params for zone id=ecced906-ad37-41c3-ad42-864c1e77463f name=test-zone3 2020-12-05T19:21:41.694 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:21:41.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:21:41.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:21:41.696 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:21:41.772 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:21:41.773 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:21:42.030 INFO:teuthology.orchestra.run.smithi023.stderr:Sending period to new master zone e622b2e3-1a91-4dc1-b31e-43ede659e70f 2020-12-05T19:21:43.137 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","epoch":3,"predecessor_uuid":"88dc4e82-8825-4cad-908f-99aa84fb6061","sync_status":["","1_1607195878.275808_415.1","",""],"period_map":{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f"},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":4} 2020-12-05T19:21:43.137 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "e622b2e3-1a91-4dc1-b31e-43ede659e70f" 2020-12-05T19:22:43.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:22:43.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:22:43.484 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:22:43.524 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:22:43.524 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:22:43.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:22:43.549+0000 7f3e75efc080 0 failed reading obj info from .rgw.root:zone_info.e622b2e3-1a91-4dc1-b31e-43ede659e70f: (2) No such file or directory 2020-12-05T19:22:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:22:43.549+0000 7f3e75efc080 0 WARNING: could not read zone params for zone id=e622b2e3-1a91-4dc1-b31e-43ede659e70f name=test-zone1 2020-12-05T19:22:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:22:43.553+0000 7f3e75efc080 0 failed reading obj info from .rgw.root:zone_info.ecced906-ad37-41c3-ad42-864c1e77463f: (2) No such file or directory 2020-12-05T19:22:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:22:43.553+0000 7f3e75efc080 0 WARNING: could not read zone params for zone id=ecced906-ad37-41c3-ad42-864c1e77463f name=test-zone3 2020-12-05T19:22:43.572 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:22:43.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:22:43.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:22:43.574 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:22:43.653 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:22:43.654 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:22:43.900 INFO:teuthology.orchestra.run.smithi023.stderr:Sending period to new master zone e622b2e3-1a91-4dc1-b31e-43ede659e70f 2020-12-05T19:22:45.006 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","epoch":4,"predecessor_uuid":"88dc4e82-8825-4cad-908f-99aa84fb6061","sync_status":["","1_1607195878.275808_415.1","",""],"period_map":{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f"},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":4} 2020-12-05T19:22:45.006 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-12-05T19:23:45.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:23:45.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:23:45.009 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:23:45.049 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:23:45.049 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:23:45.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:23:45.072+0000 7f075ca71080 0 failed reading obj info from .rgw.root:zone_info.e622b2e3-1a91-4dc1-b31e-43ede659e70f: (2) No such file or directory 2020-12-05T19:23:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:23:45.072+0000 7f075ca71080 0 WARNING: could not read zone params for zone id=e622b2e3-1a91-4dc1-b31e-43ede659e70f name=test-zone1 2020-12-05T19:23:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:23:45.072+0000 7f075ca71080 0 failed reading obj info from .rgw.root:zone_info.ecced906-ad37-41c3-ad42-864c1e77463f: (2) No such file or directory 2020-12-05T19:23:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:23:45.072+0000 7f075ca71080 0 WARNING: could not read zone params for zone id=ecced906-ad37-41c3-ad42-864c1e77463f name=test-zone3 2020-12-05T19:23:45.099 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:23:45.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:23:45.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:23:45.101 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:23:45.182 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:23:45.182 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:23:45.434 INFO:teuthology.orchestra.run.smithi023.stderr:Sending period to new master zone e622b2e3-1a91-4dc1-b31e-43ede659e70f 2020-12-05T19:23:46.545 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","epoch":5,"predecessor_uuid":"88dc4e82-8825-4cad-908f-99aa84fb6061","sync_status":["","1_1607195878.275808_415.1","",""],"period_map":{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":4} 2020-12-05T19:23:46.546 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-12-05T19:24:46.548 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-12-05T19:24:46.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:24:46.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:24:46.550 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:24:46.590 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:46.591 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:46.802 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:24:46.839 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196036.591269_315.1","last_update":"2020-12-05T19:20:36.591269Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:24:46.840 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:24:46.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:24:46.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:24:46.841 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:24:46.877 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:46.878 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:47.149 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196036.591269_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:20:36.591269Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:24:47.150 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:24:47.150 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:24:47.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:24:47.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:24:47.153 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:24:47.195 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:47.195 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:47.446 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196036.591269_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:20:36.591269Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:24:47.446 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:24:47.447 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:24:47.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:24:47.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:24:47.448 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:24:47.488 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:47.488 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:47.763 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196036.591269_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:20:36.591269Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:24:47.764 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:24:47.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:47.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'NFZJ37NSJ0K1XNCM', '--secret', 'FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:47.765 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key NFZJ37NSJ0K1XNCM --secret FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:47.848 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:47.848 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:47.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:24:47.915+0000 7f5ab229f080 0 failed reading obj info from .rgw.root:zone_info.e622b2e3-1a91-4dc1-b31e-43ede659e70f: (2) No such file or directory 2020-12-05T19:24:47.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:24:47.915+0000 7f5ab229f080 0 WARNING: could not read zone params for zone id=e622b2e3-1a91-4dc1-b31e-43ede659e70f name=test-zone1 2020-12-05T19:24:47.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:24:47.915+0000 7f5ab229f080 0 failed reading obj info from .rgw.root:zone_info.ecced906-ad37-41c3-ad42-864c1e77463f: (2) No such file or directory 2020-12-05T19:24:47.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:24:47.915+0000 7f5ab229f080 0 WARNING: could not read zone params for zone id=ecced906-ad37-41c3-ad42-864c1e77463f name=test-zone3 2020-12-05T19:24:47.930 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"04ace84e-8211-4b37-a54d-d756017e5630","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"NFZJ37NSJ0K1XNCM","secret_key":"FCFqArnWhgMgFHoGVwE8TBWazCH2lmfO"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:24:47.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:47.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:47.932 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:48.015 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:48.016 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:55.919 INFO:teuthology.orchestra.run.smithi023.stderr:Sending period to new master zone e622b2e3-1a91-4dc1-b31e-43ede659e70f 2020-12-05T19:24:57.029 INFO:teuthology.orchestra.run.smithi023.stdout:{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","epoch":6,"predecessor_uuid":"88dc4e82-8825-4cad-908f-99aa84fb6061","sync_status":["","1_1607195878.275808_415.1","",""],"period_map":{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"04ace84e-8211-4b37-a54d-d756017e5630","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"04ace84e-8211-4b37-a54d-d756017e5630","val":4102370268},{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":4} 2020-12-05T19:24:57.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.031 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:57.070 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:57.071 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:57.094 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:24:57.085+0000 7f050a413080 0 failed reading obj info from .rgw.root:zone_info.3ee88d97-6118-4d9f-87f0-34d91092c64f: (2) No such file or directory 2020-12-05T19:24:57.094 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:24:57.085+0000 7f050a413080 0 WARNING: could not read zone params for zone id=3ee88d97-6118-4d9f-87f0-34d91092c64f name=test-zone2 2020-12-05T19:24:57.095 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:24:57.085+0000 7f050a413080 0 failed reading obj info from .rgw.root:zone_info.90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d: (2) No such file or directory 2020-12-05T19:24:57.095 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:24:57.085+0000 7f050a413080 0 WARNING: could not read zone params for zone id=90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d name=test-zone4 2020-12-05T19:24:57.105 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}} 2020-12-05T19:24:57.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.107 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:57.190 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:57.190 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:57.209 INFO:teuthology.orchestra.run.smithi185.stderr:zone name remove not found in zonegroup test-zonegroup 2020-12-05T19:24:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-12-05T19:24:57.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.217 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:57.253 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:57.254 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:24:57.299+0000 7f9a62efe080 0 failed reading obj info from .rgw.root:zone_info.e622b2e3-1a91-4dc1-b31e-43ede659e70f: (2) No such file or directory 2020-12-05T19:24:57.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:24:57.299+0000 7f9a62efe080 0 WARNING: could not read zone params for zone id=e622b2e3-1a91-4dc1-b31e-43ede659e70f name=test-zone1 2020-12-05T19:24:57.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:24:57.299+0000 7f9a62efe080 0 failed reading obj info from .rgw.root:zone_info.ecced906-ad37-41c3-ad42-864c1e77463f: (2) No such file or directory 2020-12-05T19:24:57.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:24:57.299+0000 7f9a62efe080 0 WARNING: could not read zone params for zone id=ecced906-ad37-41c3-ad42-864c1e77463f name=test-zone3 2020-12-05T19:24:57.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-12-05T19:24:57.307+0000 7f9a62efe080 0 zone id 04ace84e-8211-4b37-a54d-d756017e5630 is not a part of zonegroup default 2020-12-05T19:24:57.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.326 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:57.366 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:57.366 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:57.431 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:24:57.421+0000 7fb7fe19a080 0 WARNING: can't generate connection for zone 04ace84e-8211-4b37-a54d-d756017e5630 id remove: no endpoints defined 2020-12-05T19:24:57.638 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:24:57.631+0000 7f4455ffb700 -1 failed to decode period 2020-12-05T19:24:57.639 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:24:57.631+0000 7f6dfe7fc700 -1 failed to decode period 2020-12-05T19:24:57.639 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:24:57.629+0000 7f7e54ff9700 -1 failed to decode period 2020-12-05T19:24:57.679 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","epoch":7,"predecessor_uuid":"88dc4e82-8825-4cad-908f-99aa84fb6061","sync_status":["","1_1607195878.275808_415.1","",""],"period_map":{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":4} 2020-12-05T19:24:57.679 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-12-05T19:24:57.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.681 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:57.759 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:57.760 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:57.793 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"c4eac3f8-5e14-40b0-bbc5-3f486d62f531","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}} 2020-12-05T19:24:57.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.794 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:57.875 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:57.876 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:57.944 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"5b43ee1b-0f5f-400c-9a6a-c78bb5bd6eaa","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab"} 2020-12-05T19:24:57.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:57.946 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:58.025 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:58.025 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:58.314 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:24:58.307+0000 7f6dfe7fc700 -1 failed to decode period 2020-12-05T19:24:58.315 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:24:58.307+0000 7f4455ffb700 -1 failed to decode period 2020-12-05T19:24:58.315 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:24:58.305+0000 7f7e54ff9700 -1 failed to decode period 2020-12-05T19:24:58.345 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","epoch":8,"predecessor_uuid":"88dc4e82-8825-4cad-908f-99aa84fb6061","sync_status":["","1_1607195878.275808_415.1","",""],"period_map":{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","zonegroups":[{"id":"c4eac3f8-5e14-40b0-bbc5-3f486d62f531","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"5b43ee1b-0f5f-400c-9a6a-c78bb5bd6eaa","zones":[{"id":"5b43ee1b-0f5f-400c-9a6a-c78bb5bd6eaa","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"5b43ee1b-0f5f-400c-9a6a-c78bb5bd6eaa","val":729652598},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":4} 2020-12-05T19:24:58.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:58.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:58.347 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:58.430 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:58.430 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:58.482 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:24:58.473+0000 7f038a876080 0 zone id 5b43ee1b-0f5f-400c-9a6a-c78bb5bd6eaa is not a part of zonegroup test-zonegroup 2020-12-05T19:24:58.483 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:24:58.473+0000 7f038a876080 0 zone id 5b43ee1b-0f5f-400c-9a6a-c78bb5bd6eaa is not a part of zonegroup default 2020-12-05T19:24:58.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:58.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:58.503 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:58.580 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:58.581 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:58.628 INFO:teuthology.orchestra.run.smithi185.stderr:2020-12-05T19:24:58.617+0000 7fd1c0ee9080 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-12-05T19:24:58.634 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-12-05T19:24:58.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:58.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:58.635 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:58.711 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:58.711 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:58.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:58.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:24:58.745 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:24:58.828 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:24:58.828 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:24:59.039 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:24:59.035+0000 7f4455ffb700 -1 failed to decode period 2020-12-05T19:24:59.040 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:24:59.035+0000 7f6dfe7fc700 -1 failed to decode period 2020-12-05T19:24:59.064 INFO:teuthology.orchestra.run.smithi185.stdout:{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","epoch":9,"predecessor_uuid":"88dc4e82-8825-4cad-908f-99aa84fb6061","sync_status":["","1_1607195878.275808_415.1","",""],"period_map":{"id":"99a897cd-76db-49ef-bcce-15cfc0f426ad","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","zones":[{"id":"3ee88d97-6118-4d9f-87f0-34d91092c64f","name":"test-zone2","endpoints":["http://smithi023.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","name":"test-zone4","endpoints":["http://smithi023.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","name":"test-zone1","endpoints":["http://smithi185.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ecced906-ad37-41c3-ad42-864c1e77463f","name":"test-zone3","endpoints":["http://smithi185.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"3ee88d97-6118-4d9f-87f0-34d91092c64f","val":2774224536},{"key":"90aef8bc-2d96-4660-bd9c-8e2ad8ff7a8d","val":1120231748},{"key":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","val":1713834403},{"key":"ecced906-ad37-41c3-ad42-864c1e77463f","val":891893117}]},"master_zonegroup":"test-zonegroup","master_zone":"e622b2e3-1a91-4dc1-b31e-43ede659e70f","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"4217a5cb-deb1-4741-a893-a42d4fc447ab","realm_name":"test-realm","realm_epoch":4} 2020-12-05T19:24:59.065 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-12-05T19:24:59.066 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-50 2020-12-05T19:24:59.180 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:24:59.169+0000 7f7e54ff9700 -1 failed to decode period 2020-12-05T19:24:59.550 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-51 2020-12-05T19:24:59.627 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-52 2020-12-05T19:24:59.691 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-53 2020-12-05T19:24:59.731 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-54 2020-12-05T19:24:59.807 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-55 2020-12-05T19:25:02.990 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-12-05T19:25:02.991 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-56 2020-12-05T19:25:03.605 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-57 2020-12-05T19:25:04.294 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-58 2020-12-05T19:25:05.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tazcyr-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:25:05.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tazcyr-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:25:05.325 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tazcyr-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:25:05.360 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:05.361 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:05.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tazcyr-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:25:05.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tazcyr-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:25:05.691 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tazcyr-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:25:05.773 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:05.774 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:06.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tazcyr-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:25:06.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tazcyr-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:25:06.083 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tazcyr-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:25:06.168 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:06.168 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:06.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:06.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:06.489 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:06.562 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:06.562 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:06.833 INFO:teuthology.orchestra.run.smithi185.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18-tazcyr-9","obj_delete_at_hint.0000000026","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32:8","data.full-sync.index.3ee88d97-6118-4d9f-87f0-34d91092c64f.3","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49-tazcyr-40","obj_delete_at_hint.0000000028","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47:10","data.full-sync.index.ecced906-ad37-41c3-ad42-864c1e77463f.3","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-47:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4:8","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30:8","obj_delete_at_hint.0000000015","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46-tazcyr-37","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","data.full-sync.index.3ee88d97-6118-4d9f-87f0-34d91092c64f.1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:1","data.full-sync.index.ecced906-ad37-41c3-ad42-864c1e77463f.2","data_log.3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38-tazcyr-29","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-46:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3:8","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:3","2020-12-05-19-3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1-tazcyr-42","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35-tazcyr-26","obj_delete_at_hint.0000000076","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23:1","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32:8","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-48:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5:8","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:7","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:7","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5-tazcyr-5","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1-tazcyr-44","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45:10","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:7","obj_delete_at_hint.0000000008","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.5-tazcyr-54","2020-12-05-19-3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2-tazcyr-43","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.7-tazcyr-56","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40-tazcyr-31","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16-tazcyr-7","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1-tazcyr-1","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.2.retry","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31:8","obj_delete_at_hint.0000000041","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:3","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.0","obj_delete_at_hint.0000000018","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.2-tazcyr-51","obj_delete_at_hint.0000000092","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44-tazcyr-35","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20-tazcyr-11","sync.error-log.4","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:7","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:1","obj_delete_at_hint.0000000014","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-48:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25-tazcyr-16","obj_delete_at_hint.0000000071","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.9-tazcyr-58","obj_delete_at_hint.0000000074","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27-tazcyr-18","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45:10","obj_delete_at_hint.0000000121","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:3","data.full-sync.index.3ee88d97-6118-4d9f-87f0-34d91092c64f.2","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50-tazcyr-41","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:3","obj_delete_at_hint.0000000032","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31-tazcyr-22","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22:1","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.0.retry","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:1","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29:8","obj_delete_at_hint.0000000053","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.39-tazcyr-30","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47-tazcyr-38","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.4-tazcyr-53","sync.error-log.3","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:8","meta.log.99a897cd-76db-49ef-bcce-15cfc0f426ad.1","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23-tazcyr-14","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:1","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-46:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3:8","mdlog.sync-status.shard.2","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24:1","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5-tazcyr-48","obj_delete_at_hint.0000000045","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34:8","obj_delete_at_hint.0000000002","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:3","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26-tazcyr-17","obj_delete_at_hint.0000000116","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28-tazcyr-19","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34-tazcyr-25","obj_delete_at_hint.0000000034","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:8","obj_delete_at_hint.0000000101","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:5","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.1","mdlog.sync-status.shard.1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:7","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22:1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:5","obj_delete_at_hint.0000000066","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.3","obj_delete_at_hint.0000000068","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24-tazcyr-15","obj_delete_at_hint.0000000085","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3-tazcyr-3","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.3.retry","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2-tazcyr-2","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.ecced906-ad37-41c3-ad42-864c1e77463f.0","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46:10","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3-tazcyr-46","obj_delete_at_hint.0000000100","datalog.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32-tazcyr-23","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:1","obj_delete_at_hint.0000000093","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43-tazcyr-34","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.8-tazcyr-57","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-49:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4-tazcyr-4","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.3-tazcyr-52","sync.error-log.2","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:1","data_log.0","obj_delete_at_hint.0000000012","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.3.retry","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.1.retry","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48-tazcyr-39","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.42-tazcyr-33","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-47:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41-tazcyr-32","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46:10","obj_delete_at_hint.0000000020","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:5","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","sync.error-log.1","obj_delete_at_hint.0000000033","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4-tazcyr-47","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.1-tazcyr-50","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:8","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24:1","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.1","obj_delete_at_hint.0000000108","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47:10","obj_delete_at_hint.0000000094","data.full-sync.index.ecced906-ad37-41c3-ad42-864c1e77463f.1","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.2","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21-tazcyr-12","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30:8","obj_delete_at_hint.0000000051","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.6-tazcyr-55","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:8","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37-tazcyr-28","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.2","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23:1","obj_delete_at_hint.0000000046","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29-tazcyr-20","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:3","meta.full-sync.index.2","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22-tazcyr-13","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45-tazcyr-36","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1-tazcyr-49","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17-tazcyr-8","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.1.retry","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:5","data_log.2","obj_delete_at_hint.0000000067","data.full-sync.index.3ee88d97-6118-4d9f-87f0-34d91092c64f.0","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:8","obj_delete_at_hint.0000000086","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:3","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.0","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33-tazcyr-24","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-49:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30-tazcyr-21","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:7","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19-tazcyr-10","datalog.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:1","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.0.retry","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:1","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.2.retry","obj_delete_at_hint.0000000124","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:5","obj_delete_at_hint.0000000001","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2-tazcyr-45","obj_delete_at_hint.0000000126","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:5","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:8","obj_delete_at_hint.0000000055","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.3","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35:8","obj_delete_at_hint.0000000042","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.36-tazcyr-27","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6-tazcyr-6","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:1"] 2020-12-05T19:25:06.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:06.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:06.864 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:06.901 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:06.901 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:07.169 INFO:teuthology.orchestra.run.smithi023.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","datalog.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18-tazcyr-9","obj_delete_at_hint.0000000026","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49-tazcyr-40","obj_delete_at_hint.0000000028","data.full-sync.index.ecced906-ad37-41c3-ad42-864c1e77463f.3","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-47:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4:8","obj_delete_at_hint.0000000040","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.2.retry","meta.full-sync.index.0","obj_delete_at_hint.0000000015","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46-tazcyr-37","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:1","data.full-sync.index.ecced906-ad37-41c3-ad42-864c1e77463f.2","data_log.3","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47:10","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-46:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3:8","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:3","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45:10","2020-12-05-19-3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1-tazcyr-42","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35-tazcyr-26","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:7","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5-tazcyr-5","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1-tazcyr-44","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45:10","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.3","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:7","obj_delete_at_hint.0000000008","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.5-tazcyr-54","2020-12-05-19-3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2-tazcyr-43","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.40-tazcyr-31","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16-tazcyr-7","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1-tazcyr-1","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33:8","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31:8","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.0","obj_delete_at_hint.0000000041","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.0","obj_delete_at_hint.0000000018","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.2-tazcyr-51","obj_delete_at_hint.0000000092","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20-tazcyr-11","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-46:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3:8","sync.error-log.4","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:7","obj_delete_at_hint.0000000014","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-48:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5:8","obj_delete_at_hint.0000000112","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.3","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25-tazcyr-16","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27-tazcyr-18","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.0.retry","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","obj_delete_at_hint.0000000125","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:7","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50-tazcyr-41","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31-tazcyr-22","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22:1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47-tazcyr-38","sync.error-log.3","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:8","meta.log.99a897cd-76db-49ef-bcce-15cfc0f426ad.1","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.3.retry","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23-tazcyr-14","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23:1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:8","mdlog.sync-status.shard.2","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24:1","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5-tazcyr-48","obj_delete_at_hint.0000000045","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34:8","obj_delete_at_hint.0000000002","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26-tazcyr-17","obj_delete_at_hint.0000000116","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:1","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28-tazcyr-19","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34-tazcyr-25","obj_delete_at_hint.0000000034","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:8","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:7","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22:1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:5","obj_delete_at_hint.0000000066","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:5","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.3","obj_delete_at_hint.0000000068","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24-tazcyr-15","obj_delete_at_hint.0000000085","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3-tazcyr-3","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:1","meta.full-sync.index.3","obj_delete_at_hint.0000000118","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.2","obj_delete_at_hint.0000000036","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35:8","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2-tazcyr-2","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.ecced906-ad37-41c3-ad42-864c1e77463f.0","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.2","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3-tazcyr-46","obj_delete_at_hint.0000000100","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32-tazcyr-23","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:1","obj_delete_at_hint.0000000093","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.1","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.43-tazcyr-34","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.0","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.8-tazcyr-57","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4-tazcyr-4","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:5","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46:10","sync.error-log.2","data_log.0","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34:8","obj_delete_at_hint.0000000012","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.3.retry","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.1.retry","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48-tazcyr-39","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46:10","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","sync.error-log.1","obj_delete_at_hint.0000000033","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4-tazcyr-47","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:1","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.1","obj_delete_at_hint.0000000108","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:3","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47:10","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-49:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1:5","obj_delete_at_hint.0000000094","data.full-sync.index.ecced906-ad37-41c3-ad42-864c1e77463f.1","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:1","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.2","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21-tazcyr-12","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30:8","obj_delete_at_hint.0000000051","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-48:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5:8","obj_delete_at_hint.0000000106","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:7","obj_delete_at_hint.0000000017","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:8","obj_delete_at_hint.0000000023","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24:1","obj_delete_at_hint.0000000107","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.37-tazcyr-28","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23:1","obj_delete_at_hint.0000000046","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29-tazcyr-20","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:3","meta.full-sync.index.2","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22-tazcyr-13","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45-tazcyr-36","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1-tazcyr-49","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17-tazcyr-8","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:5","data_log.2","obj_delete_at_hint.0000000067","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:3","obj_delete_at_hint.0000000086","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:3","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:3","sync.error-log.5","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-47:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4:8","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33-tazcyr-24","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:8","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:1","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-49:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1:5","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:1","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.1.retry","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30-tazcyr-21","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19-tazcyr-10","datalog.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.0.retry","datalog.sync-status.shard.ecced906-ad37-41c3-ad42-864c1e77463f.2.retry","obj_delete_at_hint.0000000124","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:5","obj_delete_at_hint.0000000001","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2-tazcyr-45","obj_delete_at_hint.0000000126","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:5","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:3","obj_delete_at_hint.0000000055","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:3","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6-tazcyr-6","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.ecced906-ad37-41c3-ad42-864c1e77463f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:1"] 2020-12-05T19:25:07.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:07.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:07.200 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:07.240 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:07.240 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:07.518 INFO:teuthology.orchestra.run.smithi185.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","datalog.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.18-tazcyr-9","obj_delete_at_hint.0000000026","data.full-sync.index.3ee88d97-6118-4d9f-87f0-34d91092c64f.3","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.49-tazcyr-40","obj_delete_at_hint.0000000028","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47:10","obj_delete_at_hint.0000000040","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.2.retry","meta.full-sync.index.0","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30:8","obj_delete_at_hint.0000000015","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46-tazcyr-37","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","data.full-sync.index.3ee88d97-6118-4d9f-87f0-34d91092c64f.1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:1","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:1","data_log.3","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47:10","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.38-tazcyr-29","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45:10","2020-12-05-19-3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.1-tazcyr-42","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35-tazcyr-26","obj_delete_at_hint.0000000076","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23:1","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32:8","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-48:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:7","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.5-tazcyr-5","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.1-tazcyr-44","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.3","obj_delete_at_hint.0000000008","2020-12-05-19-3ee88d97-6118-4d9f-87f0-34d91092c64f.5466.2-tazcyr-43","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.16-tazcyr-7","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.1-tazcyr-1","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.2.retry","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33:8","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.0","obj_delete_at_hint.0000000041","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:3","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.44-tazcyr-35","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20-tazcyr-11","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-46:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:7","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:7","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.3","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25-tazcyr-16","obj_delete_at_hint.0000000071","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.9-tazcyr-58","obj_delete_at_hint.0000000074","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27-tazcyr-18","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.0.retry","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45:10","obj_delete_at_hint.0000000121","data_log.1","obj_delete_at_hint.0000000125","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:7","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:3","data.full-sync.index.3ee88d97-6118-4d9f-87f0-34d91092c64f.2","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.50-tazcyr-41","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:3","obj_delete_at_hint.0000000032","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31-tazcyr-22","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22:1","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.0.retry","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22:1","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29:8","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47-tazcyr-38","meta.log.99a897cd-76db-49ef-bcce-15cfc0f426ad.1","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.3.retry","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23-tazcyr-14","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23:1","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-46:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:8","mdlog.sync-status.shard.2","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5-tazcyr-48","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26-tazcyr-17","obj_delete_at_hint.0000000116","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:1","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28-tazcyr-19","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34-tazcyr-25","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:5","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.1","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:5","meta.log.cb156068-23b3-4eca-8803-e19dbeb79445.1","meta.log.cb156068-23b3-4eca-8803-e19dbeb79445.0","obj_delete_at_hint.0000000068","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24-tazcyr-15","obj_delete_at_hint.0000000085","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:3","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.3-tazcyr-3","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.3.retry","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:1","meta.full-sync.index.3","obj_delete_at_hint.0000000118","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.2","obj_delete_at_hint.0000000036","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35:8","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.2-tazcyr-2","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.2","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46:10","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3-tazcyr-46","obj_delete_at_hint.0000000100","datalog.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32-tazcyr-23","obj_delete_at_hint.0000000093","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.1","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.0","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-49:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.4-tazcyr-4","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.3-tazcyr-52","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46:10","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:1","data_log.0","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34:8","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.48-tazcyr-39","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-47:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.41-tazcyr-32","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33:8","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:5","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4-tazcyr-47","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:3","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:1","meta.log.88dc4e82-8825-4cad-908f-99aa84fb6061.1","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24:1","obj_delete_at_hint.0000000108","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:3","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-49:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1:5","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:1","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21-tazcyr-12","obj_delete_at_hint.0000000051","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.6-tazcyr-55","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-48:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5:8","obj_delete_at_hint.0000000106","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:7","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24:1","obj_delete_at_hint.0000000107","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.2","obj_delete_at_hint.0000000046","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29-tazcyr-20","meta.full-sync.index.2","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22-tazcyr-13","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45-tazcyr-36","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1-tazcyr-49","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.17-tazcyr-8","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.1.retry","data_log.2","obj_delete_at_hint.0000000067","data.full-sync.index.3ee88d97-6118-4d9f-87f0-34d91092c64f.0","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:3","obj_delete_at_hint.0000000086","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:3","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.0","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-47:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4:8","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33-tazcyr-24","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:1","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.1.retry","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30-tazcyr-21","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:7","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19-tazcyr-10","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:1","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:1","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.2-tazcyr-45","obj_delete_at_hint.0000000126","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:3","obj_delete_at_hint.0000000055","datalog.sync-status.shard.3ee88d97-6118-4d9f-87f0-34d91092c64f.3","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:3","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","bucket.sync-status.3ee88d97-6118-4d9f-87f0-34d91092c64f:tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35:8","obj_delete_at_hint.0000000042","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:5","2020-12-05-19-e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.6-tazcyr-6","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-12-05T19:25:07.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:07.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:07.550 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:07.589 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:07.589 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:07.827 INFO:teuthology.orchestra.run.smithi023.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","datalog.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.2.retry","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-38:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.47:10","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-36:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.45:10","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.3","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-24:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.33:8","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.0","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:5","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-46:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.3:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:7","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.3","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.0.retry","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","obj_delete_at_hint.0000000125","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:7","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:8","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-13:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.22:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","meta.log.99a897cd-76db-49ef-bcce-15cfc0f426ad.1","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.3.retry","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-14:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.23:1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-23:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.32:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:8","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-22:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.31:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:5","meta.log.cb156068-23b3-4eca-8803-e19dbeb79445.1","meta.log.cb156068-23b3-4eca-8803-e19dbeb79445.0","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-20:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.29:8","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:1","meta.full-sync.index.3","obj_delete_at_hint.0000000118","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.2","obj_delete_at_hint.0000000036","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-26:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.35:8","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.2","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","obj_delete_at_hint.0000000100","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.1","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-21:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.30:8","obj_delete_at_hint.0000000093","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.1","data.full-sync.index.e622b2e3-1a91-4dc1-b31e-43ede659e70f.0","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-19:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.28:5","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-37:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.46:10","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-25:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.34:8","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-16:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.25:1","meta.log.88dc4e82-8825-4cad-908f-99aa84fb6061.1","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:3","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-49:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6054.1:5","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:1","obj_delete_at_hint.0000000051","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-48:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.5:8","obj_delete_at_hint.0000000106","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-12:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.21:7","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-15:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.24:1","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","obj_delete_at_hint.0000000067","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:3","obj_delete_at_hint.0000000086","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:3","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-47:e622b2e3-1a91-4dc1-b31e-43ede659e70f.5864.4:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-11:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.20:8","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-17:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.26:1","datalog.sync-status.shard.e622b2e3-1a91-4dc1-b31e-43ede659e70f.1.retry","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-10:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.19:3","obj_delete_at_hint.0000000055","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:3","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-18:e622b2e3-1a91-4dc1-b31e-43ede659e70f.4346.27:5","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-12-05T19:25:07.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:07.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:07.871 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:07.911 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:07.911 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:08.154 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196306.448837_227.3","last_update":"2020-12-05T19:25:06.448837Z"},{"marker":"1_1607196306.448837_459.3","last_update":"2020-12-05T19:25:06.448837Z"},{"marker":"1_1607196306.448837_345.3","last_update":"2020-12-05T19:25:06.448837Z"},{"marker":"1_1607196306.448837_320.2","last_update":"2020-12-05T19:25:06.448837Z"}] 2020-12-05T19:25:08.155 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:25:08.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:08.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:08.156 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:08.198 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:08.199 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:08.437 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:08.438 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:25:13.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:13.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:13.441 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:13.482 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:13.483 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:13.742 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:13.743 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:25:18.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:18.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:18.746 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:18.788 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:18.789 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:19.031 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:19.032 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:25:24.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:24.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:24.035 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:24.079 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:24.079 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:24.340 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:24.340 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:25:29.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:29.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:29.343 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:29.648 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:29.648 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:25:30.557 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.544+0000 7f7f42ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.557 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.544+0000 7f7f3bfef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.557 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.544+0000 7f7f617fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.558 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.544+0000 7f7f5affd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.559 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f59ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.559 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f597fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.559 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f39feb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.560 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f427fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.560 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f5a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.560 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f60ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.560 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f437fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.561 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f437fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.561 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f3e7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.561 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f417fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.561 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f3e7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.561 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f3f7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.562 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f3afed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.562 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f40ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.562 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f3d7f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.562 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f3bfef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.563 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f407f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:30.563 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:25:30.548+0000 7f7f3d7f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:25:34.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:34.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:34.651 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:34.695 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:34.954 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:34.955 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:25:39.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:39.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:39.958 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:40.257 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:40.257 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:25:45.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:45.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:45.260 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:45.302 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:45.302 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:45.566 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:45.567 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:25:50.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:50.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:50.569 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:50.612 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:50.612 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:50.866 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:50.867 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:25:55.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:55.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:25:55.869 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:25:55.911 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:25:55.911 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:25:56.171 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:25:56.172 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:01.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:01.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:01.175 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:01.220 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:01.220 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:01.477 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:01.478 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:06.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:06.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:06.481 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:06.522 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:06.523 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:06.776 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_227.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_459.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_345.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_320.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:06.777 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:26:06.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:06.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:06.778 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:06.817 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:06.818 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:07.058 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196306.448837_227.3","last_update":"2020-12-05T19:25:06.448837Z"},{"marker":"1_1607196306.448837_459.3","last_update":"2020-12-05T19:25:06.448837Z"},{"marker":"1_1607196306.448837_345.3","last_update":"2020-12-05T19:25:06.448837Z"},{"marker":"1_1607196306.448837_320.2","last_update":"2020-12-05T19:25:06.448837Z"}] 2020-12-05T19:26:07.058 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:26:07.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:07.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:07.060 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:07.140 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:07.140 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:07.389 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10689636149165243743},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_227.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_459.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_345.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_320.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:07.390 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:26:07.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:07.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:07.392 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:07.471 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:07.471 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:07.731 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196306.448837_227.3","last_update":"2020-12-05T19:25:06.448837Z"},{"marker":"1_1607196306.448837_459.3","last_update":"2020-12-05T19:25:06.448837Z"},{"marker":"1_1607196306.448837_345.3","last_update":"2020-12-05T19:25:06.448837Z"},{"marker":"1_1607196306.448837_320.2","last_update":"2020-12-05T19:25:06.448837Z"}] 2020-12-05T19:26:07.732 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:26:07.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:07.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:07.733 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:07.775 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:07.775 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:08.049 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:08.050 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:13.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:13.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:13.052 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:13.095 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:13.095 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:13.371 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:13.372 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:18.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:18.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:18.374 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:18.415 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:18.416 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:18.697 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:18.698 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:23.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:23.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:23.701 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:23.743 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:23.743 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:24.016 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:24.016 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:29.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:29.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:29.018 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:29.339 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:29.340 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:34.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:34.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:34.342 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:34.383 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:34.383 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:34.658 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:34.659 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:39.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:39.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:39.661 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:39.980 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:39.981 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:44.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:44.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:44.984 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:45.026 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:45.027 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:45.308 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:45.308 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:50.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:50.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:50.311 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:50.353 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:50.354 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:50.627 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:50.627 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:26:55.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:55.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:26:55.630 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:26:55.673 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:26:55.673 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:26:55.950 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:26:55.951 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:27:00.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:00.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:00.953 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:00.996 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:00.997 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:01.274 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607195965.209376_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:19:25.209376Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196164.993838_355.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:22:44.993838Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607195806.116793_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:16:46.116793Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607195580.099522_101.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:13:00.099522Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:27:01.274 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1607196306.448837_227.3 target=1_1607195965.209376_136.1, shard=1 master=1_1607196306.448837_459.3 target=1_1607196164.993838_355.1, shard=2 master=1_1607196306.448837_345.3 target=1_1607195806.116793_231.1, shard=3 master=1_1607196306.448837_320.2 target=1_1607195580.099522_101.3 2020-12-05T19:27:02.115 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ecb7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.115 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ecaff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.116 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ecd7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.116 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ecbff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.116 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ecc7f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.117 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ec87f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.117 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ecd7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.117 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6eca7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.117 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ed4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.118 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6eccff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.118 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ee0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.118 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.105+0000 7f6ee2ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.120 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ec9ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.121 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6eceffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.121 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ec97f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.121 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ece7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.121 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ed5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.122 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ecb7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.122 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ecbff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.122 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ecaff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.122 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ec8ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.122 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ec8ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.123 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ecd7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.123 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ecc7f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.124 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ed4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.124 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ec6fed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.124 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6eca7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.125 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ec6fed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.125 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ee0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.125 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ee0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.125 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6eccff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.125 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6ee0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:02.126 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:27:02.109+0000 7f6eccff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:27:06.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:06.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:06.277 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:06.320 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:06.321 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:06.598 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_227.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_459.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_345.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196306.448837_320.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:06.448837Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:27:06.599 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:27:06.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:06.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:06.600 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:06.679 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:06.679 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:06.917 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607196336.160972_203.1","last_update":"2020-12-05T19:25:36.160972Z"},{"marker":"1_1607196336.163558_446.1","last_update":"2020-12-05T19:25:36.163558Z"},{"marker":"1_1607196336.166805_329.1","last_update":"2020-12-05T19:25:36.166805Z"},{"marker":"1_1607196336.158584_329.1","last_update":"2020-12-05T19:25:36.158584Z"}] 2020-12-05T19:27:06.917 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:27:06.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:06.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:06.919 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:06.964 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:06.964 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:07.217 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9760760649708652305},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196336.160972_203.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.160972Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196336.163558_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.163558Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196336.166805_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.166805Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196336.158584_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.158584Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:27:07.217 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:27:07.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:07.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:07.219 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:07.259 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:07.259 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:07.520 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607196336.160972_203.1","last_update":"2020-12-05T19:25:36.160972Z"},{"marker":"1_1607196336.163558_446.1","last_update":"2020-12-05T19:25:36.163558Z"},{"marker":"1_1607196336.166805_329.1","last_update":"2020-12-05T19:25:36.166805Z"},{"marker":"1_1607196336.158584_329.1","last_update":"2020-12-05T19:25:36.158584Z"}] 2020-12-05T19:27:07.520 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:27:07.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:07.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:07.522 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:07.564 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:07.564 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:07.817 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3761623777010785199},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196336.160972_203.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.160972Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196336.163558_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.163558Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196336.166805_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.166805Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196336.158584_329.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.158584Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:27:07.818 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:27:07.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:07.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:07.820 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:07.900 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:07.900 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:08.152 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196334.746381_194.1","last_update":"2020-12-05T19:25:34.746381Z"},{"marker":"1_1607196334.748373_372.1","last_update":"2020-12-05T19:25:34.748373Z"},{"marker":"1_1607196334.750511_210.1","last_update":"2020-12-05T19:25:34.750511Z"},{"marker":"1_1607196334.744395_263.1","last_update":"2020-12-05T19:25:34.744395Z"}] 2020-12-05T19:27:08.152 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:27:08.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:08.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:08.154 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:08.232 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:08.233 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:08.495 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4132799784717954996},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196334.746381_194.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.746381Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196334.748373_372.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.748373Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196334.750511_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.750511Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196334.744395_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.744395Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:27:08.496 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:27:08.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:08.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:08.497 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:08.576 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:08.576 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:08.827 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196334.746381_194.1","last_update":"2020-12-05T19:25:34.746381Z"},{"marker":"1_1607196334.748373_372.1","last_update":"2020-12-05T19:25:34.748373Z"},{"marker":"1_1607196334.750511_210.1","last_update":"2020-12-05T19:25:34.750511Z"},{"marker":"1_1607196334.744395_263.1","last_update":"2020-12-05T19:25:34.744395Z"}] 2020-12-05T19:27:08.828 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:27:08.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:08.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:08.829 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:08.873 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:08.873 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:09.123 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6833204601607133234},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196334.746381_194.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.746381Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196334.748373_372.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.748373Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196334.750511_210.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.750511Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196334.744395_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.744395Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:27:09.124 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:27:09.125 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2020-12-05T19:27:09.125 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-59 2020-12-05T19:27:09.169 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-60 2020-12-05T19:27:09.239 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-61 2020-12-05T19:27:09.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:09.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:09.891 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:09.932 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:09.932 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:10.196 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.26.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:10.196 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:27:10.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:10.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:10.198 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:10.243 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:10.243 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:10.466 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.26.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:10.466 WARNING:rgw_multi.tests:bucket tazcyr-59 zone test-zone2 behind zone test-zone1: shard=2 master=00000000001.45.5 target=, shard=7 master=00000000001.53.5 target= 2020-12-05T19:27:15.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:15.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:15.469 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:15.513 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:15.513 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:15.794 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:15.794 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:27:15.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:15.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:15.796 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:15.835 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:15.836 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:16.090 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.26.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:16.091 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:27:16.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:16.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:16.092 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:16.174 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:16.175 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:16.443 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:16.444 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:27:16.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:16.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:16.446 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:16.526 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:16.526 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:16.785 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.26.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:16.786 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:27:16.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:16.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:16.788 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:16.835 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:16.836 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:17.110 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.26.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:17.111 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:27:17.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:17.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:17.112 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:17.191 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:17.191 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:17.463 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.28.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:17.464 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-59 2020-12-05T19:27:17.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:17.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:17.466 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:17.511 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:17.512 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:17.803 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:17.803 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-59 2020-12-05T19:27:17.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:17.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:17.805 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:17.845 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:17.845 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:18.107 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.28.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:18.108 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-59 2020-12-05T19:27:18.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:18.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:18.109 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:18.153 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:18.154 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:18.420 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:18.421 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-59 2020-12-05T19:27:18.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:18.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:18.423 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:18.503 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:18.503 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:18.757 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.41.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.63.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:18.757 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-59 2020-12-05T19:27:18.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:18.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:18.759 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:18.842 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:18.842 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:19.116 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:19.116 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-59 2020-12-05T19:27:19.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:19.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:19.118 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:19.198 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:19.198 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:19.438 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.41.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.63.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:19.438 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-59 2020-12-05T19:27:19.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:19.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:19.440 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:19.757 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:19.758 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-59 2020-12-05T19:27:19.758 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-59 zones={test-zone2, test-zone1} 2020-12-05T19:27:21.189 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:27:21.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:21.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:21.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:21.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:21.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:27:21.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:21.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:21.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:21.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:21.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:21.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:21.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:21.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:21.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:21.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:21.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:21.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:21.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:22.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:22.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:22.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:22.335 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-59 zones={test-zone2, test-zone1} 2020-12-05T19:27:22.335 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-59 zones={test-zone3, test-zone1} 2020-12-05T19:27:23.242 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:27:23.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:23.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:23.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:23.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:23.250 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:27:23.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:23.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:23.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:23.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:23.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:23.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:23.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:23.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:23.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:23.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:23.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:23.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:23.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:24.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:24.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:24.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:24.387 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-59 zones={test-zone3, test-zone1} 2020-12-05T19:27:24.387 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-59 zones={test-zone3, test-zone2} 2020-12-05T19:27:25.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:27:25.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:25.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:25.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:25.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:25.366 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:27:25.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:25.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:25.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:25.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:25.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:25.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:25.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:25.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:25.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:25.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:25.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:25.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:26.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:26.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:26.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:26.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:26.477 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-59 zones={test-zone3, test-zone2} 2020-12-05T19:27:26.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:26.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:26.479 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:26.521 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:26.521 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:26.786 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.72.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:26.787 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:27:26.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:26.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:26.789 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:26.831 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:26.831 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:27.130 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:27.131 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:27:27.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:27.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:27.132 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:27.174 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:27.174 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:27.442 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.72.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:27.443 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:27:27.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:27.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:27.444 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:27.526 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:27.526 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:27.790 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:27.791 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:27:27.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:27.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:27.792 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:27.871 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:27.872 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:28.119 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.48.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.72.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:28.119 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:27:28.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:28.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:28.121 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:28.164 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:28.164 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:28.448 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.48.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.72.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.55.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:28.448 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:27:28.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:28.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:28.450 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:28.533 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:28.533 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:28.795 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.46.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.55.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.41.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:28.796 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-60 2020-12-05T19:27:28.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:28.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:28.797 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:28.842 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:28.843 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:29.123 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:29.124 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-60 2020-12-05T19:27:29.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:29.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:29.125 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:29.170 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:29.170 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:29.445 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.46.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.55.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.41.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:29.445 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-60 2020-12-05T19:27:29.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:29.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:29.447 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:29.490 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:29.491 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:29.758 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:29.759 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-60 2020-12-05T19:27:29.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:29.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:29.760 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:29.841 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:29.842 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:30.073 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.83.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.56.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:30.073 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-60 2020-12-05T19:27:30.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:30.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:30.075 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:30.160 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:30.160 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:30.425 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:30.426 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-60 2020-12-05T19:27:30.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:30.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:30.427 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:30.511 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:30.511 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:30.740 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.83.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.56.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:30.741 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-60 2020-12-05T19:27:30.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:30.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:30.742 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:30.785 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:30.786 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:31.063 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:31.064 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-60 2020-12-05T19:27:31.065 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-60 zones={test-zone2, test-zone1} 2020-12-05T19:27:32.758 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:27:32.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:32.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:32.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:32.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:32.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:27:32.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:32.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:32.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:32.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:32.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:32.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:32.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:33.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:33.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:33.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:33.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:33.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:33.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:33.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:33.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:33.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:33.898 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-60 zones={test-zone2, test-zone1} 2020-12-05T19:27:33.899 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-60 zones={test-zone3, test-zone1} 2020-12-05T19:27:34.732 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:27:34.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:34.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:34.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:34.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:34.740 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:27:34.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:34.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:34.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:34.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:34.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:34.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:34.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:35.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:35.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:35.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:35.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:35.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:35.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:35.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:35.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:35.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:35.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-60 zones={test-zone3, test-zone1} 2020-12-05T19:27:35.837 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-60 zones={test-zone3, test-zone2} 2020-12-05T19:27:36.343 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:27:36.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:36.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:36.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:36.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:36.354 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:27:36.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:36.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:36.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:36.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:36.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:36.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:36.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:36.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:36.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:36.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:36.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:36.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:37.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:37.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:37.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:37.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:37.464 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-60 zones={test-zone3, test-zone2} 2020-12-05T19:27:37.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:37.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:37.466 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:37.507 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:37.508 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:37.766 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.38.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.66.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:37.767 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:27:37.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:37.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:37.768 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:37.812 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:38.099 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.66.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:38.100 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:27:38.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:38.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:38.102 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:38.143 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:38.144 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:38.391 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.38.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.66.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:38.392 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:27:38.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:38.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:38.394 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:38.474 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:38.475 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:38.744 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.66.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:38.745 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:27:38.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:38.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:38.746 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:38.830 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:38.830 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:39.067 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.38.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.66.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:39.068 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:27:39.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:39.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:39.069 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:39.114 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:39.115 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:39.394 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.36.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.66.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:39.394 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:27:39.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:39.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:39.397 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:39.772 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.31.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.32.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.35.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:39.773 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-61 2020-12-05T19:27:39.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:39.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:39.774 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:39.818 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:39.818 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:40.077 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.31.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:40.077 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-61 2020-12-05T19:27:40.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:40.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:40.079 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:40.121 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:40.384 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.31.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.32.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.35.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:40.385 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-61 2020-12-05T19:27:40.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:40.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:40.386 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:40.429 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:40.430 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:40.692 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.31.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:40.692 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-61 2020-12-05T19:27:40.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:40.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:40.694 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:40.771 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:40.772 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:41.019 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.53.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.87.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.48.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:41.020 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-61 2020-12-05T19:27:41.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:41.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:41.021 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:41.104 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:41.105 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:41.381 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.87.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:41.381 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-61 2020-12-05T19:27:41.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:41.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:41.384 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:41.463 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:41.464 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:41.712 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.53.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.87.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.48.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:27:41.713 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-61 2020-12-05T19:27:41.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:41.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:41.714 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:42.037 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.87.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:27:42.038 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-61 2020-12-05T19:27:42.038 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-61 zones={test-zone2, test-zone1} 2020-12-05T19:27:43.094 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:27:43.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:43.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:43.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:43.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:43.103 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:27:43.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:43.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:43.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:43.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:43.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:43.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:43.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:43.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:43.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:43.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:43.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:43.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:43.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:43.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:43.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:44.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:44.203 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-61 zones={test-zone2, test-zone1} 2020-12-05T19:27:44.203 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-61 zones={test-zone3, test-zone1} 2020-12-05T19:27:44.696 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:27:44.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:44.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:44.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:44.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:44.703 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:27:44.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:44.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:44.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:44.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:44.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:44.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:44.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:44.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:45.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:45.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:45.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:45.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:45.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:45.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:45.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:45.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:45.795 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-61 zones={test-zone3, test-zone1} 2020-12-05T19:27:45.795 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-61 zones={test-zone3, test-zone2} 2020-12-05T19:27:46.652 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:27:46.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:46.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:46.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:46.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:46.664 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:27:46.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:27:46.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:27:46.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:27:46.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:27:46.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:46.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:46.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:27:46.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:47.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:47.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:27:47.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:47.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:47.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:27:47.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:47.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:47.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:27:47.805 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-61 zones={test-zone3, test-zone2} 2020-12-05T19:27:47.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tazcyr-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:47.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tazcyr-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:47.806 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tazcyr-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:27:47.847 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:47.847 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:48.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tazcyr-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:48.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tazcyr-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:48.162 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tazcyr-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:27:48.246 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:48.246 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:48.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tazcyr-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:48.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tazcyr-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:48.561 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tazcyr-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:27:48.639 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:48.640 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:48.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tazcyr-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:48.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tazcyr-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:48.950 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tazcyr-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:27:49.033 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:49.034 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:49.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tazcyr-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:49.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tazcyr-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:49.355 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tazcyr-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:27:49.437 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:49.437 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:49.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tazcyr-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:49.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tazcyr-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:27:49.755 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tazcyr-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:27:49.837 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:49.838 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:52.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:52.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:52.875 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:52.917 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:52.917 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:53.185 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.32.5"},{"key":1,"val":"00000000004.52.1"},{"key":2,"val":"00000000006.59.5"},{"key":3,"val":"00000000004.42.1"},{"key":4,"val":"00000000006.59.5"},{"key":5,"val":"00000000004.85.1"},{"key":6,"val":"00000000004.40.1"},{"key":7,"val":"00000000006.63.5"},{"key":8,"val":"00000000004.52.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-12-05T19:27:53.186 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:27:53.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:53.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:53.187 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:53.231 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:53.231 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:53.493 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.32.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.59.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.59.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.63.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-12-05T19:27:53.494 WARNING:rgw_multi.tests:bucket tazcyr-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.52.1 target=, shard=5 master=00000000004.85.1 target=, shard=9 master=00000000004.53.1 target= 2020-12-05T19:27:58.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:58.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:27:58.497 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:27:58.542 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:27:58.542 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:27:58.812 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.32.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.59.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.59.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.63.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-12-05T19:27:58.813 WARNING:rgw_multi.tests:bucket tazcyr-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.52.1 target=, shard=5 master=00000000004.85.1 target=, shard=9 master=00000000004.53.1 target= 2020-12-05T19:28:03.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:03.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:03.816 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:03.864 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:03.864 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:04.133 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.32.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.59.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.59.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.63.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-12-05T19:28:04.133 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:28:04.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:04.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:04.135 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:04.175 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:04.176 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:04.447 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.32.5"},{"key":1,"val":"00000000004.52.1"},{"key":2,"val":"00000000006.59.5"},{"key":3,"val":"00000000004.42.1"},{"key":4,"val":"00000000006.59.5"},{"key":5,"val":"00000000004.85.1"},{"key":6,"val":"00000000004.40.1"},{"key":7,"val":"00000000006.63.5"},{"key":8,"val":"00000000004.52.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-12-05T19:28:04.447 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:28:04.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:04.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:04.449 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:04.536 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:04.537 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:04.812 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.32.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.59.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.59.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.63.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-12-05T19:28:04.813 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:28:04.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:04.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:04.816 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:04.897 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:04.898 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:05.139 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.32.5"},{"key":1,"val":"00000000004.52.1"},{"key":2,"val":"00000000006.59.5"},{"key":3,"val":"00000000004.42.1"},{"key":4,"val":"00000000006.59.5"},{"key":5,"val":"00000000004.85.1"},{"key":6,"val":"00000000004.40.1"},{"key":7,"val":"00000000006.63.5"},{"key":8,"val":"00000000004.52.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-12-05T19:28:05.139 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:28:05.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:05.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:05.141 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:05.189 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:05.189 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:05.475 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.32.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.59.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.59.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.63.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-12-05T19:28:05.475 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-59 2020-12-05T19:28:05.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:05.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:05.477 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:05.518 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:05.518 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:05.747 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000002.32.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.55.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.82.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:05.747 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-59 2020-12-05T19:28:05.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:05.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:05.749 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:05.792 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:05.793 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:06.066 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.32.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.50.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.55.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.82.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:06.067 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-59 2020-12-05T19:28:06.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:06.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:06.068 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:06.111 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:06.111 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:06.366 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000002.32.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.55.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.82.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:06.367 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-59 2020-12-05T19:28:06.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:06.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:06.369 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:06.413 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:06.413 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:06.680 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.32.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.50.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.55.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.82.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:06.680 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-59 2020-12-05T19:28:06.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:06.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:06.681 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:06.719 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:06.719 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:06.965 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000002.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.69.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:06.966 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-59 2020-12-05T19:28:06.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:06.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:06.968 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:07.050 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:07.050 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:07.318 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.69.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.73.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:07.319 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-59 2020-12-05T19:28:07.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:07.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:07.321 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:07.404 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:07.404 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:07.652 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000002.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.69.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:07.653 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-59 2020-12-05T19:28:07.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:07.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:07.654 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:07.697 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:07.698 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:07.953 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.69.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.73.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:07.954 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-59 2020-12-05T19:28:07.954 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-59 zones={test-zone2, test-zone1} 2020-12-05T19:28:09.377 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:09.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:09.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:09.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:09.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:09.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:09.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:09.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:09.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:09.388 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:09.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:09.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:09.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:09.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:09.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:09.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:09.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:09.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:09.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:09.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:09.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:09.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:09.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:09.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:09.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:10.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:10.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:10.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:10.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:10.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:10.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:10.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:10.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:10.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:10.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:10.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:11.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:11.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:11.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:11.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:11.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:11.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:11.655 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-59 zones={test-zone2, test-zone1} 2020-12-05T19:28:11.656 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-59 zones={test-zone3, test-zone1} 2020-12-05T19:28:12.339 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:12.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:12.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:12.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:12.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:12.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:12.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:12.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:12.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:12.348 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:12.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:12.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:12.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:12.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:12.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:12.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:12.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:12.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:12.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:12.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:12.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:12.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:12.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:12.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:12.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:12.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:13.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:13.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:13.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:13.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:13.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:13.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:13.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:13.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:13.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:13.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:13.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:14.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:14.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:14.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:14.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:14.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:14.538 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-59 zones={test-zone3, test-zone1} 2020-12-05T19:28:14.539 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-59 zones={test-zone3, test-zone2} 2020-12-05T19:28:15.546 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:15.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:15.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:15.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:15.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:15.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:15.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:15.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:15.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:15.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:15.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:15.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:15.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:15.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:15.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:15.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:15.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:15.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:15.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:15.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:15.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:15.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:15.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:16.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:16.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:16.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:16.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:16.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:16.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:16.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:16.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:16.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:16.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:16.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:17.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:17.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:17.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:17.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:17.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:17.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:17.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:17.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:17.801 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-59 zones={test-zone3, test-zone2} 2020-12-05T19:28:17.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:17.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:17.803 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:17.843 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:17.843 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:18.082 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.62.5"},{"key":1,"val":"00000000004.61.1"},{"key":2,"val":"00000000006.53.5"},{"key":3,"val":"00000000004.57.1"},{"key":4,"val":"00000000006.86.5"},{"key":5,"val":"00000000004.60.1"},{"key":6,"val":"00000000004.44.1"},{"key":7,"val":"00000000006.65.5"},{"key":8,"val":"00000000004.45.1"},{"key":9,"val":"00000000004.70.1"},{"key":10,"val":"00000000004.41.1"}]} 2020-12-05T19:28:18.082 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:28:18.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:18.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:18.084 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:18.129 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:18.129 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:18.431 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.62.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.53.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.86.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.65.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}}] 2020-12-05T19:28:18.432 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:28:18.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:18.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:18.433 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:18.474 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:18.475 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:18.735 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.62.5"},{"key":1,"val":"00000000004.61.1"},{"key":2,"val":"00000000006.53.5"},{"key":3,"val":"00000000004.57.1"},{"key":4,"val":"00000000006.86.5"},{"key":5,"val":"00000000004.60.1"},{"key":6,"val":"00000000004.44.1"},{"key":7,"val":"00000000006.65.5"},{"key":8,"val":"00000000004.45.1"},{"key":9,"val":"00000000004.70.1"},{"key":10,"val":"00000000004.41.1"}]} 2020-12-05T19:28:18.735 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:28:18.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:18.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:18.737 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:18.819 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:18.819 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:19.088 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.62.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.53.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.86.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.65.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}}] 2020-12-05T19:28:19.089 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:28:19.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:19.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:19.090 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:19.174 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:19.175 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:19.450 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.62.5"},{"key":1,"val":"00000000004.61.1"},{"key":2,"val":"00000000006.53.5"},{"key":3,"val":"00000000004.57.1"},{"key":4,"val":"00000000006.86.5"},{"key":5,"val":"00000000004.60.1"},{"key":6,"val":"00000000004.44.1"},{"key":7,"val":"00000000006.65.5"},{"key":8,"val":"00000000004.45.1"},{"key":9,"val":"00000000004.70.1"},{"key":10,"val":"00000000004.41.1"}]} 2020-12-05T19:28:19.451 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:28:19.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:19.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:19.452 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:19.498 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:19.498 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:19.789 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.62.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.86.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.65.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}}] 2020-12-05T19:28:19.790 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-60 2020-12-05T19:28:19.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:19.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:19.791 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:19.876 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:19.877 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:20.120 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000002.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.66.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:20.120 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-60 2020-12-05T19:28:20.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:20.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:20.122 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:20.165 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:20.165 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:20.463 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.57.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.66.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:20.463 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-60 2020-12-05T19:28:20.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:20.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:20.465 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:20.758 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000002.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.66.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:20.759 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-60 2020-12-05T19:28:20.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:20.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:20.761 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:20.805 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:20.805 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:21.078 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.57.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.66.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:21.079 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-60 2020-12-05T19:28:21.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:21.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:21.080 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:21.119 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:21.120 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:21.379 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000002.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.91.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.62.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:21.379 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-60 2020-12-05T19:28:21.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:21.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:21.381 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:21.462 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:21.462 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:21.749 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.57.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.91.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.62.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:21.750 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-60 2020-12-05T19:28:21.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:21.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:21.751 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:21.831 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:21.831 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:22.080 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000002.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.91.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.62.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:22.081 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-60 2020-12-05T19:28:22.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:22.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:22.082 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:22.125 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:22.126 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:22.396 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.57.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.91.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.62.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:22.397 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-60 2020-12-05T19:28:22.397 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-60 zones={test-zone2, test-zone1} 2020-12-05T19:28:23.005 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:23.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:23.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:23.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:23.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:23.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:23.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:23.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:23.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:23.016 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:23.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:23.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:23.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:23.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:23.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:23.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:23.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:23.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:23.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:23.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:23.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:23.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:23.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:23.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:23.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:23.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:23.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:23.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:23.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:24.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:24.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:24.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:24.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:24.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:24.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:24.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:24.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:24.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:24.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:25.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:25.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:25.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:25.283 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-60 zones={test-zone2, test-zone1} 2020-12-05T19:28:25.283 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-60 zones={test-zone3, test-zone1} 2020-12-05T19:28:25.439 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:25.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:25.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:25.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:25.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:25.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:25.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:25.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:25.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:25.449 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:25.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:25.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:25.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:25.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:25.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:25.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:25.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:25.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:25.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:25.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:25.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:25.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:25.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:25.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:26.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:26.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:26.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:26.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:26.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:26.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:26.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:26.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:26.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:26.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:26.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:27.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:27.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:27.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:27.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:27.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:27.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:27.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:27.725 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-60 zones={test-zone3, test-zone1} 2020-12-05T19:28:27.726 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-60 zones={test-zone3, test-zone2} 2020-12-05T19:28:28.420 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:28.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:28.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:28.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:28.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:28.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:28.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:28.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:28.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:28.433 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:28.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:28.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:28.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:28.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:28.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:28.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:28.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:28.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:28.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:28.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:28.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:28.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:28.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:28.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:29.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:29.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:29.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:29.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:29.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:29.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:29.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:29.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:29.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:29.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:29.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:30.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:30.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:30.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:30.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:30.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:30.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:30.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:30.676 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-60 zones={test-zone3, test-zone2} 2020-12-05T19:28:30.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:30.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:30.678 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:30.720 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:30.720 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:30.965 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.48.5"},{"key":1,"val":"00000000004.57.1"},{"key":2,"val":"00000000006.50.5"},{"key":3,"val":"00000000004.61.1"},{"key":4,"val":"00000000006.76.5"},{"key":5,"val":"00000000004.83.1"},{"key":6,"val":"00000000004.84.1"},{"key":7,"val":"00000000006.55.5"},{"key":8,"val":"00000000004.74.1"},{"key":9,"val":"00000000004.63.1"},{"key":10,"val":"00000000004.65.1"}]} 2020-12-05T19:28:30.966 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:28:30.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:30.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:30.967 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:31.011 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:31.279 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.76.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.83.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.55.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1"}}}] 2020-12-05T19:28:31.280 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:28:31.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:31.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:31.282 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:31.323 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:31.324 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:31.592 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.48.5"},{"key":1,"val":"00000000004.57.1"},{"key":2,"val":"00000000006.50.5"},{"key":3,"val":"00000000004.61.1"},{"key":4,"val":"00000000006.76.5"},{"key":5,"val":"00000000004.83.1"},{"key":6,"val":"00000000004.84.1"},{"key":7,"val":"00000000006.55.5"},{"key":8,"val":"00000000004.74.1"},{"key":9,"val":"00000000004.63.1"},{"key":10,"val":"00000000004.65.1"}]} 2020-12-05T19:28:31.592 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:28:31.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:31.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:31.594 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:31.680 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:31.680 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.76.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.83.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.55.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1"}}}] 2020-12-05T19:28:31.941 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:28:31.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:31.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:31.942 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:31.981 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:31.982 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:32.249 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.48.5"},{"key":1,"val":"00000000004.57.1"},{"key":2,"val":"00000000006.50.5"},{"key":3,"val":"00000000004.61.1"},{"key":4,"val":"00000000006.76.5"},{"key":5,"val":"00000000004.83.1"},{"key":6,"val":"00000000004.84.1"},{"key":7,"val":"00000000006.55.5"},{"key":8,"val":"00000000004.74.1"},{"key":9,"val":"00000000004.63.1"},{"key":10,"val":"00000000004.65.1"}]} 2020-12-05T19:28:32.249 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:28:32.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:32.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:32.251 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:32.572 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.48.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.50.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.76.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.83.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.55.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1"}}}] 2020-12-05T19:28:32.572 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-61 2020-12-05T19:28:32.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:32.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:32.574 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:32.614 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:32.615 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:32.876 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000002.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.40.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.59.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:32.877 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-61 2020-12-05T19:28:32.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:32.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:32.878 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:32.922 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:32.923 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:33.193 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.38.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.40.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:33.194 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-61 2020-12-05T19:28:33.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:33.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:33.195 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:33.237 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:33.518 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000002.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.40.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.59.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:33.519 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-61 2020-12-05T19:28:33.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:33.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:33.520 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:33.564 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:33.564 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:33.835 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.38.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.40.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:33.835 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-61 2020-12-05T19:28:33.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:33.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:33.837 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:33.925 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:33.926 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:34.145 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000002.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.95.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:34.145 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-61 2020-12-05T19:28:34.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:34.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:34.147 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:34.189 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:34.189 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:34.469 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.63.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.61.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.95.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:34.469 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-61 2020-12-05T19:28:34.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:34.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:34.471 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:34.551 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:34.552 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:34.800 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000002.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.95.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:34.801 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-61 2020-12-05T19:28:34.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:34.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:34.802 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:34.846 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:34.846 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:35.129 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.63.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.61.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.95.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:35.130 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-61 2020-12-05T19:28:35.130 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-61 zones={test-zone2, test-zone1} 2020-12-05T19:28:35.910 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:35.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:35.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:35.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:35.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:35.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:35.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:35.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:35.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:35.921 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:35.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:35.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:35.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:35.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:35.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:35.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:35.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:35.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:35.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:35.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:36.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:36.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:36.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:36.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:36.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:36.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:36.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:36.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:36.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:36.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:37.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:37.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:37.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:37.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:37.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:37.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:37.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:37.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:37.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:37.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:37.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:38.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:38.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-61 zones={test-zone2, test-zone1} 2020-12-05T19:28:38.156 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-61 zones={test-zone3, test-zone1} 2020-12-05T19:28:38.373 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:38.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:38.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:38.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:38.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:38.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:38.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:38.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:38.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:38.381 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:38.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:38.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:38.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:38.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:38.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:38.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:38.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:38.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:38.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:38.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:38.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:38.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:38.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:38.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:38.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:39.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:39.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:39.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:39.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:39.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:39.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:39.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:39.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:39.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:39.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:39.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:40.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:40.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:40.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:40.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:40.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:40.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:40.630 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-61 zones={test-zone3, test-zone1} 2020-12-05T19:28:40.630 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-61 zones={test-zone3, test-zone2} 2020-12-05T19:28:40.697 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:40.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:40.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:40.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:40.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:40.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:40.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:40.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:40.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:40.708 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:40.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:40.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:40.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:40.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:40.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:28:40.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:28:40.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:28:40.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:28:40.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:40.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:40.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:40.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:41.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:41.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:41.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:41.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:41.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:41.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:41.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:41.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:41.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:41.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:42.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:28:42.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:42.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:42.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:28:42.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:42.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:42.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:28:42.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:42.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:42.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:28:42.978 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-61 zones={test-zone3, test-zone2} 2020-12-05T19:28:42.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:42.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:42.980 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:43.018 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:43.019 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:43.286 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196473.195224_266.1","last_update":"2020-12-05T19:27:53.195224Z"},{"marker":"1_1607196470.109439_517.3","last_update":"2020-12-05T19:27:50.109439Z"},{"marker":"1_1607196474.164675_406.1","last_update":"2020-12-05T19:27:54.164675Z"},{"marker":"1_1607196474.164866_371.1","last_update":"2020-12-05T19:27:54.164866Z"}] 2020-12-05T19:28:43.286 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:28:43.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:43.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:43.288 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:43.332 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:43.333 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:43.605 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196473.195224_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:53.195224Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196470.109439_517.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:50.109439Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196474.164675_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.164675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196474.164866_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.164866Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:28:43.606 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:28:43.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:43.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:43.608 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:43.647 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:43.647 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:43.907 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196473.195224_266.1","last_update":"2020-12-05T19:27:53.195224Z"},{"marker":"1_1607196470.109439_517.3","last_update":"2020-12-05T19:27:50.109439Z"},{"marker":"1_1607196474.164675_406.1","last_update":"2020-12-05T19:27:54.164675Z"},{"marker":"1_1607196474.164866_371.1","last_update":"2020-12-05T19:27:54.164866Z"}] 2020-12-05T19:28:43.907 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:28:43.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:43.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:43.909 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:43.991 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:43.991 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:44.239 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10689636149165243743},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196473.195224_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:53.195224Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196470.109439_517.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:50.109439Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196474.164675_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.164675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196474.164866_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.164866Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:28:44.239 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:28:44.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:44.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:44.241 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:44.319 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:44.319 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:44.590 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196473.195224_266.1","last_update":"2020-12-05T19:27:53.195224Z"},{"marker":"1_1607196470.109439_517.3","last_update":"2020-12-05T19:27:50.109439Z"},{"marker":"1_1607196474.164675_406.1","last_update":"2020-12-05T19:27:54.164675Z"},{"marker":"1_1607196474.164866_371.1","last_update":"2020-12-05T19:27:54.164866Z"}] 2020-12-05T19:28:44.591 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:28:44.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:44.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:44.593 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:44.919 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196473.195224_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:53.195224Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196470.109439_517.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:50.109439Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196474.164675_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.164675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196474.164866_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.164866Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:28:44.919 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:28:44.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:44.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:44.921 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:45.259 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607196473.192768_221.1","last_update":"2020-12-05T19:27:53.192768Z"},{"marker":"1_1607196336.163558_446.1","last_update":"2020-12-05T19:25:36.163558Z"},{"marker":"1_1607196474.202838_365.1","last_update":"2020-12-05T19:27:54.202838Z"},{"marker":"1_1607196474.160312_372.1","last_update":"2020-12-05T19:27:54.160312Z"}] 2020-12-05T19:28:45.259 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:28:45.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:45.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:45.261 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:45.303 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:45.304 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:45.566 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9760760649708652305},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196473.192768_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:53.192768Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196336.163558_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.163558Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196474.202838_365.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.202838Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196474.160312_372.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.160312Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:28:45.567 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:28:45.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:45.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:45.568 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:45.608 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:45.608 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:45.854 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607196473.192768_221.1","last_update":"2020-12-05T19:27:53.192768Z"},{"marker":"1_1607196336.163558_446.1","last_update":"2020-12-05T19:25:36.163558Z"},{"marker":"1_1607196474.202838_365.1","last_update":"2020-12-05T19:27:54.202838Z"},{"marker":"1_1607196474.160312_372.1","last_update":"2020-12-05T19:27:54.160312Z"}] 2020-12-05T19:28:45.855 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:28:45.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:45.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:45.856 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:45.898 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:45.899 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:46.161 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3761623777010785199},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196473.192768_221.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:53.192768Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196336.163558_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:36.163558Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196474.202838_365.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.202838Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196474.160312_372.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:54.160312Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:28:46.161 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:28:46.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:46.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:46.163 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:46.244 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:46.245 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:46.506 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196472.788570_206.1","last_update":"2020-12-05T19:27:52.788570Z"},{"marker":"1_1607196334.748373_372.1","last_update":"2020-12-05T19:25:34.748373Z"},{"marker":"1_1607196473.686313_247.1","last_update":"2020-12-05T19:27:53.686313Z"},{"marker":"1_1607196472.836362_301.1","last_update":"2020-12-05T19:27:52.836362Z"}] 2020-12-05T19:28:46.507 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:28:46.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:46.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:46.508 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:46.588 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:46.588 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:46.855 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4132799784717954996},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196472.788570_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:52.788570Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196334.748373_372.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.748373Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196473.686313_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:53.686313Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196472.836362_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:52.836362Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:28:46.855 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:28:46.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:46.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:46.857 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:46.935 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:46.935 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:47.180 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196472.788570_206.1","last_update":"2020-12-05T19:27:52.788570Z"},{"marker":"1_1607196334.748373_372.1","last_update":"2020-12-05T19:25:34.748373Z"},{"marker":"1_1607196473.686313_247.1","last_update":"2020-12-05T19:27:53.686313Z"},{"marker":"1_1607196472.836362_301.1","last_update":"2020-12-05T19:27:52.836362Z"}] 2020-12-05T19:28:47.180 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:28:47.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:47.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:47.182 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:47.513 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6833204601607133234},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196472.788570_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:52.788570Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196334.748373_372.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:25:34.748373Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196473.686313_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:53.686313Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196472.836362_301.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:27:52.836362Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:28:47.513 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:28:47.514 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2020-12-05T19:28:47.515 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-62 2020-12-05T19:28:48.361 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-63 2020-12-05T19:28:48.967 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-64 2020-12-05T19:28:52.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:52.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:52.188 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:52.227 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:52.228 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:52.482 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:28:52.528 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196529.827139_512.1","last_update":"2020-12-05T19:28:49.827139Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:28:52.528 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:28:52.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:52.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:52.530 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:52.566 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:52.567 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:52.796 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196529.827139_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:28:49.827139Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:28:52.796 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:28:52.797 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:28:52.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:52.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:52.798 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:52.836 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:52.836 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:53.074 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196529.827139_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:28:49.827139Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:28:53.074 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:28:53.075 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:28:53.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:53.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:53.076 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:53.116 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:53.116 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:53.393 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196529.827139_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:28:49.827139Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:28:53.393 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:28:53.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:53.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:53.395 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:53.437 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:53.437 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:53.683 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.36.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.67.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.65.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:53.683 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:28:53.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:53.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:53.685 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:53.734 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:53.735 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:53.966 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:53.967 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:28:53.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:53.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:53.968 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:54.006 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:54.006 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:54.252 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.36.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.67.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.65.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:54.253 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:28:54.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:54.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:54.255 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:54.337 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:54.337 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:54.593 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:54.593 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:28:54.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:54.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:54.595 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:54.678 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:54.678 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:54.933 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.36.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.67.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.65.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:54.934 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:28:54.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:54.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:54.935 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:54.980 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:54.980 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:55.255 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.36.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.67.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.65.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:55.255 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:28:55.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:55.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:55.257 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:55.297 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:55.298 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:55.538 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.53.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.39.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.73.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.59.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:55.538 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-62 2020-12-05T19:28:55.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:55.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:55.539 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:55.586 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:55.586 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:55.852 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:55.852 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-62 2020-12-05T19:28:55.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:55.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:55.854 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:55.894 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:55.895 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.53.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.39.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.73.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.59.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:56.160 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-62 2020-12-05T19:28:56.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:56.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:56.162 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:56.464 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:56.464 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-62 2020-12-05T19:28:56.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:56.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:56.466 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:56.548 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:56.548 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:56.818 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.52.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.98.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.72.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:56.819 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-62 2020-12-05T19:28:56.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:56.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:56.820 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:56.901 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:56.902 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:57.178 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.98.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:57.178 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-62 2020-12-05T19:28:57.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:57.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:57.180 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:57.261 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:57.261 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:57.514 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.52.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.98.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.72.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:28:57.515 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-62 2020-12-05T19:28:57.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:57.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:28:57.516 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:28:57.560 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:28:57.561 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:28:57.821 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.98.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:28:57.822 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-62 2020-12-05T19:28:57.823 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-62 zones={test-zone2, test-zone1} 2020-12-05T19:28:58.802 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:28:58.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:58.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:58.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:58.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:58.809 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:28:58.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:28:58.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:28:58.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:28:58.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:28:58.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:58.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:58.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:28:59.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:59.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:59.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:28:59.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:59.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:59.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:28:59.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:59.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:59.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:28:59.921 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-62 zones={test-zone2, test-zone1} 2020-12-05T19:28:59.922 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-62 zones={test-zone3, test-zone1} 2020-12-05T19:29:00.140 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:29:00.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:00.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:00.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:00.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:00.147 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:29:00.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:00.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:00.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:00.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:00.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:00.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:00.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:00.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:00.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:00.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:00.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:00.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:00.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:00.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:01.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:01.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:01.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-62 zones={test-zone3, test-zone1} 2020-12-05T19:29:01.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-62 zones={test-zone3, test-zone2} 2020-12-05T19:29:01.580 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:29:01.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:01.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:01.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:01.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:01.591 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:29:01.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:01.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:01.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:01.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:01.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:01.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:01.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:01.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:01.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:02.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:02.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:02.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:02.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:02.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:02.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:02.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:02.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-62 zones={test-zone3, test-zone2} 2020-12-05T19:29:02.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:02.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:02.719 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:02.761 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:02.762 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:03.016 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.66.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.38.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:03.017 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:29:03.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:03.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:03.019 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:03.067 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:03.068 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:03.341 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.66.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:03.341 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:29:03.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:03.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:03.343 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:03.382 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:03.383 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:03.648 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.66.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.38.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:03.648 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:29:03.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:03.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:03.650 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:03.733 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:03.734 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:03.995 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.66.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:03.995 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:29:03.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:03.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:03.997 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:04.077 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:04.078 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:04.345 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.66.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.38.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:04.345 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:29:04.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:04.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:04.347 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:04.392 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:04.392 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:04.677 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.66.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:04.677 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:29:04.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:04.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:04.679 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:04.718 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:04.718 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:04.983 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.37.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.45.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:04.984 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-63 2020-12-05T19:29:04.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:04.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:04.987 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:05.030 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:05.031 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:05.308 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:05.309 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-63 2020-12-05T19:29:05.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:05.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:05.310 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:05.352 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:05.353 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:05.614 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.37.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.45.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:05.615 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-63 2020-12-05T19:29:05.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:05.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:05.616 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:05.660 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:05.661 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:05.939 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:05.940 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-63 2020-12-05T19:29:05.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:05.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:05.942 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:06.021 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:06.021 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:06.273 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.68.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:06.273 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-63 2020-12-05T19:29:06.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:06.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:06.275 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:06.361 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:06.362 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:06.647 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:06.647 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-63 2020-12-05T19:29:06.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:06.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:06.649 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:06.689 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:06.689 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:06.943 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.68.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:06.944 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-63 2020-12-05T19:29:06.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:06.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:06.945 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:06.989 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:07.266 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:07.267 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-63 2020-12-05T19:29:07.267 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-63 zones={test-zone2, test-zone1} 2020-12-05T19:29:07.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:29:07.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:07.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:07.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:07.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:07.503 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:29:07.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:07.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:07.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:07.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:07.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:07.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:07.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:07.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:07.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:07.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:08.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:08.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:08.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:08.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:08.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:08.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:08.643 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-63 zones={test-zone2, test-zone1} 2020-12-05T19:29:08.643 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-63 zones={test-zone3, test-zone1} 2020-12-05T19:29:08.731 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:29:08.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:08.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:08.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:08.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:08.739 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:29:08.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:08.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:08.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:08.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:08.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:08.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:08.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:09.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:09.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:09.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:09.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:09.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:09.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:09.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:09.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:09.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:09.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-63 zones={test-zone3, test-zone1} 2020-12-05T19:29:09.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-63 zones={test-zone3, test-zone2} 2020-12-05T19:29:10.571 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:29:10.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:10.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:10.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:10.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:10.581 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:29:10.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:10.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:10.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:10.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:10.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:10.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:10.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:10.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:10.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:11.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:11.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:11.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:11.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:11.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:11.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:11.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:11.703 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-63 zones={test-zone3, test-zone2} 2020-12-05T19:29:11.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:11.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:11.705 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:11.745 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:11.745 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:11.989 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.68.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.52.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.57.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:11.989 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:29:11.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:11.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:11.991 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:12.037 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:12.037 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:12.310 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:12.311 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:29:12.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:12.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:12.312 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:12.352 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:12.353 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:12.604 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.68.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.52.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.57.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:12.605 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:29:12.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:12.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:12.607 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:12.691 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:12.692 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:12.954 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:12.955 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:29:12.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:12.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:12.956 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:13.034 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:13.035 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:13.297 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.68.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.52.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.57.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:13.372 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:29:13.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:13.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:13.373 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:13.699 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.68.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.52.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.50.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.57.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:13.700 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:29:13.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:13.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:13.701 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:13.782 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:13.783 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:14.036 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.61.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:14.037 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-64 2020-12-05T19:29:14.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:14.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:14.038 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:14.083 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:14.083 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:14.373 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:14.374 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-64 2020-12-05T19:29:14.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:14.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:14.376 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:14.416 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:14.417 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:14.687 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.61.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:14.687 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-64 2020-12-05T19:29:14.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:14.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:14.689 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:14.732 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:14.733 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:14.983 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:14.984 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-64 2020-12-05T19:29:14.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:14.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:14.985 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:15.065 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:15.065 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:15.312 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.78.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:15.312 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-64 2020-12-05T19:29:15.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:15.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:15.314 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:15.401 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:15.401 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:15.686 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:15.686 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-64 2020-12-05T19:29:15.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:15.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:15.688 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:15.768 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:15.768 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:16.017 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.78.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:29:16.017 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-64 2020-12-05T19:29:16.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:16.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:16.019 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:16.064 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:16.064 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:16.340 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:29:16.341 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-64 2020-12-05T19:29:16.341 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-64 zones={test-zone2, test-zone1} 2020-12-05T19:29:17.935 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:29:17.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:17.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:17.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:17.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:17.947 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:29:17.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:17.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:17.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:17.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:17.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:18.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:18.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:18.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:18.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:18.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:18.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:18.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:18.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:18.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:18.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:18.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:19.088 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-64 zones={test-zone2, test-zone1} 2020-12-05T19:29:19.089 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-64 zones={test-zone3, test-zone1} 2020-12-05T19:29:19.557 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:29:19.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:19.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:19.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:19.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:19.564 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:29:19.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:19.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:19.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:19.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:19.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:19.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:19.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:19.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:19.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:20.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:20.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:20.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:20.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:20.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:20.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:20.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:20.669 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-64 zones={test-zone3, test-zone1} 2020-12-05T19:29:20.670 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-64 zones={test-zone3, test-zone2} 2020-12-05T19:29:20.853 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:29:20.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:20.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:20.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:20.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:20.868 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:29:20.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:29:20.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:29:20.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:29:20.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:29:20.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:20.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:21.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:29:21.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:21.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:21.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:29:21.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:21.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:21.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:29:21.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:21.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:21.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:29:21.986 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-64 zones={test-zone3, test-zone2} 2020-12-05T19:29:21.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tazcyr-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:21.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tazcyr-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:21.989 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tazcyr-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:29:22.031 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:22.032 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:22.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tazcyr-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:22.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tazcyr-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:22.337 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tazcyr-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:29:22.420 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:22.420 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:22.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tazcyr-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:22.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tazcyr-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:22.750 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tazcyr-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:29:22.833 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:22.833 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:23.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:23.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:23.155 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:23.232 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:23.233 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:23.454 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:29:23.491 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196563.101756_546.1","last_update":"2020-12-05T19:29:23.101756Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:29:23.492 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:29:23.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:23.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:23.493 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:23.790 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196529.827139_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:28:49.827139Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:29:23.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196563.101756_546.1 target=1_1607196529.827139_512.1 2020-12-05T19:29:28.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:28.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:28.793 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:28.830 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:28.831 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:29.088 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196529.827139_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:28:49.827139Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:29:29.089 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196563.101756_546.1 target=1_1607196529.827139_512.1 2020-12-05T19:29:34.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:34.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:34.092 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:34.134 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:34.134 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:34.397 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196563.101756_546.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:23.101756Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:29:34.398 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:29:34.398 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:29:34.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:34.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:34.400 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:34.441 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:34.441 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:34.699 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196563.101756_546.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:23.101756Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:29:34.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:29:34.700 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:29:34.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:34.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:34.701 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:34.742 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:34.742 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:35.016 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196563.101756_546.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:23.101756Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:29:35.016 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:29:37.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tazcyr-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:37.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tazcyr-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:37.512 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tazcyr-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:29:37.554 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:37.555 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:37.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tazcyr-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:37.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tazcyr-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:37.914 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tazcyr-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:29:38.007 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:38.007 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:38.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tazcyr-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:38.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tazcyr-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:29:38.312 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tazcyr-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:29:38.355 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:38.356 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:38.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:38.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:38.691 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:38.772 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:38.772 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:39.032 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.68.1"},{"key":1,"val":"00000000004.80.1"},{"key":2,"val":"00000000006.46.1"},{"key":3,"val":"00000000004.67.1"},{"key":4,"val":"00000000006.81.1"},{"key":5,"val":"00000000004.55.1"},{"key":6,"val":"00000000004.47.1"},{"key":7,"val":"00000000006.73.1"},{"key":8,"val":"00000000004.77.1"},{"key":9,"val":"00000000004.93.1"},{"key":10,"val":"00000000004.75.1"}]} 2020-12-05T19:29:39.033 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:29:39.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:39.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:39.035 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:39.080 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:39.081 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:39.356 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.68.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.81.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:29:39.357 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.80.1 target=, shard=5 master=00000000004.55.1 target=, shard=9 master=00000000004.93.1 target= 2020-12-05T19:29:44.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:44.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:44.360 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:44.404 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:44.680 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.68.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.81.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:29:44.681 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:29:44.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:44.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:44.682 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:44.722 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:44.723 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:44.987 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.68.1"},{"key":1,"val":"00000000004.80.1"},{"key":2,"val":"00000000006.46.1"},{"key":3,"val":"00000000004.67.1"},{"key":4,"val":"00000000006.81.1"},{"key":5,"val":"00000000004.55.1"},{"key":6,"val":"00000000004.47.1"},{"key":7,"val":"00000000006.73.1"},{"key":8,"val":"00000000004.77.1"},{"key":9,"val":"00000000004.93.1"},{"key":10,"val":"00000000004.75.1"}]} 2020-12-05T19:29:44.988 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:29:44.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:44.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:44.989 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:45.074 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:45.074 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:45.349 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.68.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.81.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:29:45.349 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone3 behind zone test-zone1: shard=1 master=00000000004.80.1 target=, shard=5 master=00000000004.55.1 target=, shard=9 master=00000000004.93.1 target= 2020-12-05T19:29:50.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:50.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:50.351 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:50.396 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:50.396 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:50.645 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.68.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.81.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:29:50.645 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone3 behind zone test-zone1: shard=1 master=00000000004.80.1 target=, shard=5 master=00000000004.55.1 target=, shard=9 master=00000000004.93.1 target= 2020-12-05T19:29:55.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:55.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:55.648 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:55.692 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:55.692 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:55.959 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.68.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.81.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:29:55.960 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:29:55.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:55.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:55.961 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:55.998 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:55.998 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:56.257 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.68.1"},{"key":1,"val":"00000000004.80.1"},{"key":2,"val":"00000000006.46.1"},{"key":3,"val":"00000000004.67.1"},{"key":4,"val":"00000000006.81.1"},{"key":5,"val":"00000000004.55.1"},{"key":6,"val":"00000000004.47.1"},{"key":7,"val":"00000000006.73.1"},{"key":8,"val":"00000000004.77.1"},{"key":9,"val":"00000000004.93.1"},{"key":10,"val":"00000000004.75.1"}]} 2020-12-05T19:29:56.258 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:29:56.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:56.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:29:56.259 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:29:56.306 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:29:56.307 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:29:56.585 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:29:56.586 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:01.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:01.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:01.588 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:01.634 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:01.634 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:02.021 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:02.022 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:07.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:07.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:07.024 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:07.348 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:07.348 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:12.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:12.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:12.351 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:12.396 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:12.396 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:12.670 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:12.670 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:17.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:17.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:17.672 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:17.718 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:17.719 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:18.088 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:18.088 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:23.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:23.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:23.091 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:23.137 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:23.137 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:23.417 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:23.418 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:28.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:28.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:28.420 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:28.464 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:28.465 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:28.746 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:28.747 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:33.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:33.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:33.749 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:33.792 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:33.792 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:34.068 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:34.069 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:39.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:39.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:39.071 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:39.116 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:39.116 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:39.400 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:39.401 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:44.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:44.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:44.403 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:44.448 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:44.449 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:44.728 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:44.729 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:49.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:49.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:49.731 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:50.047 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:50.047 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:30:55.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:55.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:30:55.051 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:30:55.096 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:30:55.096 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:30:55.370 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:30:55.371 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:31:00.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:00.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:00.373 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:00.422 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:00.422 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:00.700 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:31:00.701 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:31:05.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:05.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:05.704 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:05.780 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:05.781 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:06.047 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:31:06.048 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:31:11.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:11.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:11.050 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:11.374 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:31:11.375 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:31:16.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:16.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:16.377 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:16.421 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:16.421 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:16.688 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:31:16.689 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:31:21.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:21.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:21.691 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:21.736 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:21.737 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:21.999 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:31:21.999 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:31:27.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:27.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:27.002 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:27.046 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:27.047 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:27.324 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:31:27.324 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:31:32.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:32.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:32.327 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:32.373 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:32.374 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:32.648 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:31:32.648 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:31:37.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:37.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:37.651 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:37.697 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:37.697 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:37.969 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:31:37.970 WARNING:rgw_multi.tests:bucket tazcyr-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.68.1 target=00000000002.63.1, shard=4 master=00000000006.81.1 target=00000000002.69.1 2020-12-05T19:31:42.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:42.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:42.974 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:43.019 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:43.020 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:43.303 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.68.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.81.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}}] 2020-12-05T19:31:43.304 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-62 2020-12-05T19:31:43.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:43.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:43.306 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:43.348 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:43.348 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:43.610 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000004.58.1"},{"key":1,"val":"00000000002.86.1"},{"key":2,"val":"00000000004.49.1"},{"key":3,"val":"00000000002.63.1"},{"key":4,"val":"00000000004.89.1"},{"key":5,"val":"00000000002.57.1"},{"key":6,"val":"00000000002.43.1"},{"key":7,"val":"00000000004.67.1"},{"key":8,"val":"00000000002.85.1"},{"key":9,"val":"00000000002.63.1"},{"key":10,"val":"00000000002.78.1"}]} 2020-12-05T19:31:43.610 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-62 2020-12-05T19:31:43.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:43.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:43.612 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:43.658 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:43.659 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:43.940 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.58.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.89.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.43.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.67.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1"}}}] 2020-12-05T19:31:43.940 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-62 2020-12-05T19:31:43.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:43.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:43.942 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:43.983 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:43.983 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:44.245 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000004.58.1"},{"key":1,"val":"00000000002.86.1"},{"key":2,"val":"00000000004.49.1"},{"key":3,"val":"00000000002.63.1"},{"key":4,"val":"00000000004.89.1"},{"key":5,"val":"00000000002.57.1"},{"key":6,"val":"00000000002.43.1"},{"key":7,"val":"00000000004.67.1"},{"key":8,"val":"00000000002.85.1"},{"key":9,"val":"00000000002.63.1"},{"key":10,"val":"00000000002.78.1"}]} 2020-12-05T19:31:44.245 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-62 2020-12-05T19:31:44.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:44.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:44.247 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:44.291 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:44.291 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:44.560 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.58.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.89.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.43.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.67.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1"}}}] 2020-12-05T19:31:44.560 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-62 2020-12-05T19:31:44.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:44.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:44.562 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:44.648 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:44.649 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:44.903 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000004.57.1"},{"key":1,"val":"00000000002.111.1"},{"key":2,"val":"00000000004.59.1"},{"key":3,"val":"00000000002.75.1"},{"key":4,"val":"00000000004.114.1"},{"key":5,"val":"00000000002.55.1"},{"key":6,"val":"00000000002.44.1"},{"key":7,"val":"00000000004.79.1"},{"key":8,"val":"00000000002.87.1"},{"key":9,"val":"00000000002.72.1"},{"key":10,"val":"00000000002.77.1"}]} 2020-12-05T19:31:44.904 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-62 2020-12-05T19:31:44.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:44.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:44.906 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:44.985 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:44.985 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:45.273 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.57.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.111.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.59.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.114.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.79.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.87.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1"}}}] 2020-12-05T19:31:45.274 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-62 2020-12-05T19:31:45.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:45.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:45.275 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:45.357 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:45.358 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:45.617 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000004.57.1"},{"key":1,"val":"00000000002.111.1"},{"key":2,"val":"00000000004.59.1"},{"key":3,"val":"00000000002.75.1"},{"key":4,"val":"00000000004.114.1"},{"key":5,"val":"00000000002.55.1"},{"key":6,"val":"00000000002.44.1"},{"key":7,"val":"00000000004.79.1"},{"key":8,"val":"00000000002.87.1"},{"key":9,"val":"00000000002.72.1"},{"key":10,"val":"00000000002.77.1"}]} 2020-12-05T19:31:45.618 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-62 2020-12-05T19:31:45.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:45.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:45.620 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:45.666 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:45.667 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:45.935 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.57.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.111.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.59.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.114.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.79.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.87.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1"}}}] 2020-12-05T19:31:45.935 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-62 2020-12-05T19:31:45.936 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-62 zones={test-zone2, test-zone1} 2020-12-05T19:31:45.947 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:31:45.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:31:45.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:31:45.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:31:45.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:31:45.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:31:45.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:31:45.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:31:45.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:31:45.957 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:31:45.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:31:45.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:31:45.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:31:45.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:31:45.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:31:45.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:31:45.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:31:45.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:31:45.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:31:46.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:31:46.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:31:46.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:31:46.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:31:46.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:31:46.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:31:46.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:31:46.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:31:46.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:31:46.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:31:47.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:31:47.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:31:47.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:31:47.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:31:47.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:31:47.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:31:47.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:31:47.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:31:47.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:31:47.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:31:47.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:31:48.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:31:48.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:31:48.263 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-62 zones={test-zone2, test-zone1} 2020-12-05T19:31:48.264 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-62 zones={test-zone3, test-zone1} 2020-12-05T19:31:48.275 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:31:48.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:31:48.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:31:48.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:31:48.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:31:48.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:31:48.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:31:48.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:31:48.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:31:48.287 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:31:48.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:31:48.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:31:48.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:31:48.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:31:48.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:31:48.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:31:48.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:31:48.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:31:48.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:31:48.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:31:48.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:31:48.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:31:48.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:31:48.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:31:48.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:31:48.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:31:49.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:31:49.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:31:49.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:31:49.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:31:49.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:31:49.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:31:49.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:31:49.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:31:49.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:31:49.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:31:49.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:31:50.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:31:50.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:31:50.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:31:50.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:31:50.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:31:50.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-62 zones={test-zone3, test-zone1} 2020-12-05T19:31:50.511 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-62 zones={test-zone3, test-zone2} 2020-12-05T19:31:50.977 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:31:50.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:31:50.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:31:50.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:31:50.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:31:50.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:31:50.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:31:50.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:31:50.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:31:50.991 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:31:50.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:31:50.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:31:50.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:31:50.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:31:51.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:31:51.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:31:51.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:31:51.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:31:51.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:31:51.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:31:51.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:31:51.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:31:51.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:31:51.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:31:51.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:31:51.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:31:51.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:31:51.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:31:51.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:31:52.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:31:52.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:31:52.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:31:52.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:31:52.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:31:52.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:31:52.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:31:52.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:31:52.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:31:53.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:31:53.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:31:53.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:31:53.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:31:53.390 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-62 zones={test-zone3, test-zone2} 2020-12-05T19:31:53.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:53.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:53.393 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:53.435 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:53.436 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:53.715 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.59.5"},{"key":1,"val":"00000000004.84.1"},{"key":2,"val":"00000000006.78.5"},{"key":3,"val":"00000000004.46.1"},{"key":4,"val":"00000000006.50.5"},{"key":5,"val":"00000000004.58.1"},{"key":6,"val":"00000000004.66.1"},{"key":7,"val":"00000000006.64.5"},{"key":8,"val":"00000000004.49.1"},{"key":9,"val":"00000000004.85.1"},{"key":10,"val":"00000000004.37.1"}]} 2020-12-05T19:31:53.716 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:31:53.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:53.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:53.717 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:53.765 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:54.040 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.59.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.78.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}}] 2020-12-05T19:31:54.041 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:31:54.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:54.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:54.042 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:54.083 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:54.083 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:54.343 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.59.5"},{"key":1,"val":"00000000004.84.1"},{"key":2,"val":"00000000006.78.5"},{"key":3,"val":"00000000004.46.1"},{"key":4,"val":"00000000006.50.5"},{"key":5,"val":"00000000004.58.1"},{"key":6,"val":"00000000004.66.1"},{"key":7,"val":"00000000006.64.5"},{"key":8,"val":"00000000004.49.1"},{"key":9,"val":"00000000004.85.1"},{"key":10,"val":"00000000004.37.1"}]} 2020-12-05T19:31:54.343 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:31:54.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:54.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:54.346 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:54.428 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:54.429 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:54.709 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.59.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.78.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}}] 2020-12-05T19:31:54.710 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:31:54.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:54.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:54.712 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:54.751 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:54.751 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:55.001 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.59.5"},{"key":1,"val":"00000000004.84.1"},{"key":2,"val":"00000000006.78.5"},{"key":3,"val":"00000000004.46.1"},{"key":4,"val":"00000000006.50.5"},{"key":5,"val":"00000000004.58.1"},{"key":6,"val":"00000000004.66.1"},{"key":7,"val":"00000000006.64.5"},{"key":8,"val":"00000000004.49.1"},{"key":9,"val":"00000000004.85.1"},{"key":10,"val":"00000000004.37.1"}]} 2020-12-05T19:31:55.001 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:31:55.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:55.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:31:55.003 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:31:55.047 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:31:55.048 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:31:55.328 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.59.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.50.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}}] 2020-12-05T19:31:55.329 WARNING:rgw_multi.tests:bucket tazcyr-63 zone test-zone4 behind zone test-zone1: shard=2 master=00000000006.78.5 target=00000000002.69.1, shard=7 master=00000000006.64.5 target=00000000002.55.1 2020-12-05T19:32:00.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:00.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:00.331 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:00.653 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.59.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.50.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}}] 2020-12-05T19:32:00.654 WARNING:rgw_multi.tests:bucket tazcyr-63 zone test-zone4 behind zone test-zone1: shard=2 master=00000000006.78.5 target=00000000002.69.1, shard=7 master=00000000006.64.5 target=00000000002.55.1 2020-12-05T19:32:05.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:05.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:05.657 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:05.708 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:05.709 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:05.966 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.59.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.78.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.50.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.64.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.85.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}}] 2020-12-05T19:32:05.966 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-63 2020-12-05T19:32:05.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:05.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:05.968 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:06.009 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:06.009 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:06.293 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000004.56.1"},{"key":1,"val":"00000000002.84.1"},{"key":2,"val":"00000000004.72.1"},{"key":3,"val":"00000000002.44.1"},{"key":4,"val":"00000000004.48.1"},{"key":5,"val":"00000000002.55.1"},{"key":6,"val":"00000000002.58.1"},{"key":7,"val":"00000000004.52.1"},{"key":8,"val":"00000000002.56.1"},{"key":9,"val":"00000000002.85.1"},{"key":10,"val":"00000000002.36.1"}]} 2020-12-05T19:32:06.294 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-63 2020-12-05T19:32:06.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:06.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:06.297 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:06.341 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:06.342 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:06.621 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.56.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.84.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.48.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.52.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1"}}}] 2020-12-05T19:32:06.622 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-63 2020-12-05T19:32:06.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:06.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:06.624 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:06.666 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:06.920 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000004.56.1"},{"key":1,"val":"00000000002.84.1"},{"key":2,"val":"00000000004.72.1"},{"key":3,"val":"00000000002.44.1"},{"key":4,"val":"00000000004.48.1"},{"key":5,"val":"00000000002.55.1"},{"key":6,"val":"00000000002.58.1"},{"key":7,"val":"00000000004.52.1"},{"key":8,"val":"00000000002.56.1"},{"key":9,"val":"00000000002.85.1"},{"key":10,"val":"00000000002.36.1"}]} 2020-12-05T19:32:06.920 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-63 2020-12-05T19:32:06.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:06.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:06.922 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:06.967 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:06.968 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:07.239 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.56.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.84.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.48.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.52.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1"}}}] 2020-12-05T19:32:07.240 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-63 2020-12-05T19:32:07.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:07.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:07.241 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:07.323 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:07.323 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:07.570 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000004.59.5"},{"key":1,"val":"00000000002.109.1"},{"key":2,"val":"00000000004.76.5"},{"key":3,"val":"00000000002.51.1"},{"key":4,"val":"00000000004.63.5"},{"key":5,"val":"00000000002.52.1"},{"key":6,"val":"00000000002.54.1"},{"key":7,"val":"00000000004.78.5"},{"key":8,"val":"00000000002.36.1"},{"key":9,"val":"00000000002.110.1"},{"key":10,"val":"00000000002.34.1"}]} 2020-12-05T19:32:07.571 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-63 2020-12-05T19:32:07.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:07.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:07.572 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:07.655 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:07.655 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:07.949 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.59.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.109.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.76.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.63.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.78.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.110.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.34.1"}}}] 2020-12-05T19:32:07.949 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-63 2020-12-05T19:32:07.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:07.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:07.951 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:08.031 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:08.032 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:08.279 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000004.59.5"},{"key":1,"val":"00000000002.109.1"},{"key":2,"val":"00000000004.76.5"},{"key":3,"val":"00000000002.51.1"},{"key":4,"val":"00000000004.63.5"},{"key":5,"val":"00000000002.52.1"},{"key":6,"val":"00000000002.54.1"},{"key":7,"val":"00000000004.78.5"},{"key":8,"val":"00000000002.36.1"},{"key":9,"val":"00000000002.110.1"},{"key":10,"val":"00000000002.34.1"}]} 2020-12-05T19:32:08.280 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-63 2020-12-05T19:32:08.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:08.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:08.281 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:08.326 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:08.327 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:08.609 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.59.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.109.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.76.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.63.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.78.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.110.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.34.1"}}}] 2020-12-05T19:32:08.609 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-63 2020-12-05T19:32:08.610 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-63 zones={test-zone2, test-zone1} 2020-12-05T19:32:09.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:32:09.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:09.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:09.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:09.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:09.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:09.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:09.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:09.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:09.589 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:32:09.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:09.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:09.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:09.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:09.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:09.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:09.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:09.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:09.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:09.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:09.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:09.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:09.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:10.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:10.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:10.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:10.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:10.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:10.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:10.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:10.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:10.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:10.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:11.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:11.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:11.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:11.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:11.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:11.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:11.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:11.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:11.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:11.863 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-63 zones={test-zone2, test-zone1} 2020-12-05T19:32:11.863 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-63 zones={test-zone3, test-zone1} 2020-12-05T19:32:12.078 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:32:12.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:12.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:12.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:12.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:12.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:12.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:12.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:12.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:12.087 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:32:12.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:12.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:12.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:12.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:12.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:12.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:12.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:12.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:12.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:12.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:12.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:12.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:12.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:12.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:12.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:12.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:12.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:12.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:13.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:13.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:13.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:13.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:13.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:13.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:13.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:13.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:13.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:13.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:13.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:14.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:14.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:14.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:14.346 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-63 zones={test-zone3, test-zone1} 2020-12-05T19:32:14.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-63 zones={test-zone3, test-zone2} 2020-12-05T19:32:14.761 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:32:14.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:14.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:14.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:14.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:14.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:14.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:14.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:14.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:14.775 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:32:14.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:14.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:14.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:14.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:14.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:14.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:14.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:14.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:14.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:14.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:14.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:15.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:15.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:15.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:15.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:15.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:15.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:15.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:15.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:15.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:15.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:15.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:16.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:16.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:16.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:16.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:16.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:16.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:16.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:16.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:16.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:16.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:17.010 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-63 zones={test-zone3, test-zone2} 2020-12-05T19:32:17.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:17.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:17.011 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:17.056 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:17.057 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:17.318 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.80.5"},{"key":1,"val":"00000000004.73.1"},{"key":2,"val":"00000000006.62.5"},{"key":3,"val":"00000000004.75.1"},{"key":4,"val":"00000000006.64.5"},{"key":5,"val":"00000000004.58.1"},{"key":6,"val":"00000000004.95.1"},{"key":7,"val":"00000000006.67.5"},{"key":8,"val":"00000000004.69.1"},{"key":9,"val":"00000000004.65.1"},{"key":10,"val":"00000000004.62.1"}]} 2020-12-05T19:32:17.319 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:32:17.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:17.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:17.320 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:17.366 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:17.367 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:17.630 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.80.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.62.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.95.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.67.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}}] 2020-12-05T19:32:17.631 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:32:17.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:17.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:17.632 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:17.673 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:17.673 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:17.934 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.80.5"},{"key":1,"val":"00000000004.73.1"},{"key":2,"val":"00000000006.62.5"},{"key":3,"val":"00000000004.75.1"},{"key":4,"val":"00000000006.64.5"},{"key":5,"val":"00000000004.58.1"},{"key":6,"val":"00000000004.95.1"},{"key":7,"val":"00000000006.67.5"},{"key":8,"val":"00000000004.69.1"},{"key":9,"val":"00000000004.65.1"},{"key":10,"val":"00000000004.62.1"}]} 2020-12-05T19:32:17.935 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:32:17.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:17.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:17.938 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:18.019 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:18.019 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:18.288 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.80.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.62.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.95.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.67.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}}] 2020-12-05T19:32:18.288 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:32:18.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:18.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:18.290 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:18.370 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:18.370 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:18.631 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000006.80.5"},{"key":1,"val":"00000000004.73.1"},{"key":2,"val":"00000000006.62.5"},{"key":3,"val":"00000000004.75.1"},{"key":4,"val":"00000000006.64.5"},{"key":5,"val":"00000000004.58.1"},{"key":6,"val":"00000000004.95.1"},{"key":7,"val":"00000000006.67.5"},{"key":8,"val":"00000000004.69.1"},{"key":9,"val":"00000000004.65.1"},{"key":10,"val":"00000000004.62.1"}]} 2020-12-05T19:32:18.632 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:32:18.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:18.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:18.633 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:18.676 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:18.677 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:18.949 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.80.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.62.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.64.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.95.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.67.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}}] 2020-12-05T19:32:18.950 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-64 2020-12-05T19:32:18.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:18.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:18.951 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:19.033 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:19.034 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:19.294 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000004.76.5"},{"key":1,"val":"00000000002.45.1"},{"key":2,"val":"00000000004.57.5"},{"key":3,"val":"00000000002.66.1"},{"key":4,"val":"00000000004.55.5"},{"key":5,"val":"00000000002.50.1"},{"key":6,"val":"00000000002.60.1"},{"key":7,"val":"00000000004.71.5"},{"key":8,"val":"00000000002.60.1"},{"key":9,"val":"00000000002.67.1"},{"key":10,"val":"00000000002.53.1"}]} 2020-12-05T19:32:19.294 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-64 2020-12-05T19:32:19.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:19.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:19.296 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:19.357 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:19.358 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:19.636 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.76.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.57.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.55.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.71.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1"}}}] 2020-12-05T19:32:19.637 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-64 2020-12-05T19:32:19.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:19.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:19.640 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:19.925 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":"00000000004.76.5"},{"key":1,"val":"00000000002.45.1"},{"key":2,"val":"00000000004.57.5"},{"key":3,"val":"00000000002.66.1"},{"key":4,"val":"00000000004.55.5"},{"key":5,"val":"00000000002.50.1"},{"key":6,"val":"00000000002.60.1"},{"key":7,"val":"00000000004.71.5"},{"key":8,"val":"00000000002.60.1"},{"key":9,"val":"00000000002.67.1"},{"key":10,"val":"00000000002.53.1"}]} 2020-12-05T19:32:19.926 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-64 2020-12-05T19:32:19.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:19.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:19.927 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:19.964 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:19.965 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:20.223 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.76.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.57.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.55.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.71.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1"}}}] 2020-12-05T19:32:20.224 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-64 2020-12-05T19:32:20.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:20.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:20.226 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:20.306 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:20.306 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:20.562 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000004.72.1"},{"key":1,"val":"00000000002.47.1"},{"key":2,"val":"00000000004.60.1"},{"key":3,"val":"00000000002.80.1"},{"key":4,"val":"00000000004.61.1"},{"key":5,"val":"00000000002.53.1"},{"key":6,"val":"00000000002.75.1"},{"key":7,"val":"00000000004.85.1"},{"key":8,"val":"00000000002.78.1"},{"key":9,"val":"00000000002.86.1"},{"key":10,"val":"00000000002.74.1"}]} 2020-12-05T19:32:20.563 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-64 2020-12-05T19:32:20.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:20.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:20.564 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:20.646 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:20.647 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:20.919 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.60.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.85.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1"}}}] 2020-12-05T19:32:20.920 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-64 2020-12-05T19:32:20.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:20.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:20.921 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:21.004 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:21.004 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:21.260 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":"00000000004.72.1"},{"key":1,"val":"00000000002.47.1"},{"key":2,"val":"00000000004.60.1"},{"key":3,"val":"00000000002.80.1"},{"key":4,"val":"00000000004.61.1"},{"key":5,"val":"00000000002.53.1"},{"key":6,"val":"00000000002.75.1"},{"key":7,"val":"00000000004.85.1"},{"key":8,"val":"00000000002.78.1"},{"key":9,"val":"00000000002.86.1"},{"key":10,"val":"00000000002.74.1"}]} 2020-12-05T19:32:21.261 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-64 2020-12-05T19:32:21.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:21.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:21.264 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:21.310 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:21.311 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:21.580 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.60.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.85.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1"}}}] 2020-12-05T19:32:21.580 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-64 2020-12-05T19:32:21.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-64 zones={test-zone2, test-zone1} 2020-12-05T19:32:21.872 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:32:21.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:21.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:21.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:21.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:21.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:21.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:21.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:21.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:21.886 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:32:21.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:21.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:21.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:21.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:21.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:21.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:21.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:21.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:21.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:21.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:22.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:22.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:22.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:22.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:22.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:22.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:22.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:22.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:22.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:22.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:23.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:23.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:23.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:23.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:23.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:23.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:23.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:23.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:23.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:23.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:23.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:24.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:24.224 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-64 zones={test-zone2, test-zone1} 2020-12-05T19:32:24.224 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-64 zones={test-zone3, test-zone1} 2020-12-05T19:32:25.140 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:32:25.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:25.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:25.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:25.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:25.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:25.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:25.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:25.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:25.160 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:32:25.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:25.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:25.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:25.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:25.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:25.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:25.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:25.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:25.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:25.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:25.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:25.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:25.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:25.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:25.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:25.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:26.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:26.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:26.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:26.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:26.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:26.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:26.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:26.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:26.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:26.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:27.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:27.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:27.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:27.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:27.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:27.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:27.642 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-64 zones={test-zone3, test-zone1} 2020-12-05T19:32:27.643 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-64 zones={test-zone3, test-zone2} 2020-12-05T19:32:28.107 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:32:28.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:28.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:28.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:28.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:28.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:28.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:28.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:28.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:28.121 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:32:28.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-12-05T19:32:28.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-12-05T19:32:28.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-12-05T19:32:28.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-12-05T19:32:28.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-12-05T19:32:28.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-12-05T19:32:28.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-12-05T19:32:28.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-12-05T19:32:28.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:28.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:28.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-12-05T19:32:28.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:28.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:28.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-12-05T19:32:28.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:28.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:28.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-12-05T19:32:29.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:29.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:29.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-12-05T19:32:29.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:29.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:29.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-12-05T19:32:29.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:29.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:29.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-12-05T19:32:29.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:29.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:30.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-12-05T19:32:30.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:30.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:30.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-12-05T19:32:30.417 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-64 zones={test-zone3, test-zone2} 2020-12-05T19:32:30.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:30.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:30.419 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:30.460 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:30.461 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:30.733 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196581.573307_378.1","last_update":"2020-12-05T19:29:41.573307Z"},{"marker":"1_1607196578.649347_589.3","last_update":"2020-12-05T19:29:38.649347Z"},{"marker":"1_1607196581.500519_512.1","last_update":"2020-12-05T19:29:41.500519Z"},{"marker":"1_1607196581.494948_548.1","last_update":"2020-12-05T19:29:41.494948Z"}] 2020-12-05T19:32:30.733 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:32:30.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:30.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:30.735 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:30.778 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:30.779 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:31.034 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11265368414822143621},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196581.573307_378.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:41.573307Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196578.649347_589.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:38.649347Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196581.500519_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:41.500519Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196581.494948_548.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:41.494948Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:32:31.035 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-12-05T19:32:31.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:31.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:31.037 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:31.075 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:31.076 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:31.331 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196581.573307_378.1","last_update":"2020-12-05T19:29:41.573307Z"},{"marker":"1_1607196578.649347_589.3","last_update":"2020-12-05T19:29:38.649347Z"},{"marker":"1_1607196581.500519_512.1","last_update":"2020-12-05T19:29:41.500519Z"},{"marker":"1_1607196581.494948_548.1","last_update":"2020-12-05T19:29:41.494948Z"}] 2020-12-05T19:32:31.332 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:32:31.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:31.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:31.334 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:31.416 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:31.417 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:31.677 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10689636149165243743},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196581.573307_378.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:41.573307Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196578.649347_589.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:38.649347Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196581.500519_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:41.500519Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196581.494948_548.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:41.494948Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:32:31.677 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-12-05T19:32:31.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:31.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:31.679 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:31.758 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:31.759 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:32.004 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196581.573307_378.1","last_update":"2020-12-05T19:29:41.573307Z"},{"marker":"1_1607196578.649347_589.3","last_update":"2020-12-05T19:29:38.649347Z"},{"marker":"1_1607196581.500519_512.1","last_update":"2020-12-05T19:29:41.500519Z"},{"marker":"1_1607196581.494948_548.1","last_update":"2020-12-05T19:29:41.494948Z"}] 2020-12-05T19:32:32.005 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:32:32.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:32.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:32.006 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:32.049 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:32.050 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:32.329 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":958635629456111788},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196581.573307_378.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:41.573307Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196578.649347_589.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:38.649347Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196581.500519_512.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:41.500519Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196581.494948_548.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:41.494948Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:32:32.329 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-12-05T19:32:32.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:32.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:32.331 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:32.407 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:32.407 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:32.669 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607196587.207473_307.9","last_update":"2020-12-05T19:29:47.207473Z"},{"marker":"1_1607196587.207473_528.9","last_update":"2020-12-05T19:29:47.207473Z"},{"marker":"1_1607196587.207473_450.9","last_update":"2020-12-05T19:29:47.207473Z"},{"marker":"1_1607196587.207473_576.6","last_update":"2020-12-05T19:29:47.207473Z"}] 2020-12-05T19:32:32.669 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:32:32.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:32.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:32.672 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:32.714 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:32.715 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:32.976 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9760760649708652305},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196587.207473_307.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:47.207473Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196587.207473_528.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:47.207473Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196587.207473_450.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:47.207473Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196587.207473_576.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:47.207473Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:32:32.976 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-12-05T19:32:32.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:32.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:32.979 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:33.022 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:33.023 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:33.284 INFO:teuthology.orchestra.run.smithi023.stdout:[{"marker":"1_1607196587.207473_307.9","last_update":"2020-12-05T19:29:47.207473Z"},{"marker":"1_1607196587.207473_528.9","last_update":"2020-12-05T19:29:47.207473Z"},{"marker":"1_1607196587.207473_450.9","last_update":"2020-12-05T19:29:47.207473Z"},{"marker":"1_1607196587.207473_576.6","last_update":"2020-12-05T19:29:47.207473Z"}] 2020-12-05T19:32:33.285 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:32:33.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:33.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:33.287 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:33.330 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:33.330 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:33.593 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3761623777010785199},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196587.207473_307.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:47.207473Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196587.207473_528.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:47.207473Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196587.207473_450.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:47.207473Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196587.207473_576.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:47.207473Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:32:33.594 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-12-05T19:32:33.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:33.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:33.595 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:33.678 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:33.678 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:33.943 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196585.064667_277.9","last_update":"2020-12-05T19:29:45.064667Z"},{"marker":"1_1607196585.064667_457.9","last_update":"2020-12-05T19:29:45.064667Z"},{"marker":"1_1607196585.064667_314.9","last_update":"2020-12-05T19:29:45.064667Z"},{"marker":"1_1607196585.064667_500.6","last_update":"2020-12-05T19:29:45.064667Z"}] 2020-12-05T19:32:33.944 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:32:33.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:33.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:33.946 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:34.029 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:34.029 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:34.281 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4132799784717954996},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196585.064667_277.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:45.064667Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196585.064667_457.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:45.064667Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196585.064667_314.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:45.064667Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196585.064667_500.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:45.064667Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:32:34.282 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-12-05T19:32:34.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:34.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:34.283 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:34.360 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:34.361 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:34.623 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"1_1607196585.064667_277.9","last_update":"2020-12-05T19:29:45.064667Z"},{"marker":"1_1607196585.064667_457.9","last_update":"2020-12-05T19:29:45.064667Z"},{"marker":"1_1607196585.064667_314.9","last_update":"2020-12-05T19:29:45.064667Z"},{"marker":"1_1607196585.064667_500.6","last_update":"2020-12-05T19:29:45.064667Z"}] 2020-12-05T19:32:34.624 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:32:34.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:34.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:34.625 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:34.668 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:34.668 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:34.942 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6833204601607133234},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1607196585.064667_277.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:45.064667Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1607196585.064667_457.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:45.064667Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1607196585.064667_314.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:45.064667Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1607196585.064667_500.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:29:45.064667Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-12-05T19:32:34.942 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-12-05T19:32:34.943 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2020-12-05T19:32:34.944 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-65 2020-12-05T19:32:35.192 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tazcyr-66 2020-12-05T19:32:36.141 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tazcyr-67 2020-12-05T19:32:38.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:38.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:38.695 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:38.738 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:38.739 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:39.056 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.101.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:32:39.056 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-65 2020-12-05T19:32:39.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:39.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:39.058 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:39.102 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:39.102 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:39.384 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:32:39.384 WARNING:rgw_multi.tests:bucket tazcyr-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.101.5 target= 2020-12-05T19:32:44.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:44.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:44.387 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:44.431 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:44.432 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:44.715 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000011.101.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:32:44.715 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-65 2020-12-05T19:32:44.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:44.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:44.717 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:44.757 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:44.757 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:45.015 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.101.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:32:45.015 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-65 2020-12-05T19:32:45.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:45.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:45.017 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:45.097 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:45.098 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:45.377 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000011.101.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:32:45.378 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-65 2020-12-05T19:32:45.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:45.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:45.381 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:45.459 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:45.460 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:45.730 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.101.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:32:45.731 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-65 2020-12-05T19:32:45.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:45.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:45.732 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:45.780 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:46.054 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.101.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:32:46.055 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-65 2020-12-05T19:32:46.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:46.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:46.056 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:46.095 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:46.095 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:46.354 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.69.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:32:46.355 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-65 2020-12-05T19:32:46.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:46.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:46.356 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:46.402 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:46.402 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:46.693 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:32:46.694 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-65 2020-12-05T19:32:46.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:46.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:46.697 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:46.740 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:46.740 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:46.989 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.69.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:32:46.990 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-65 2020-12-05T19:32:46.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:46.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:46.991 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:47.035 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:47.036 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:47.313 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:32:47.314 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-65 2020-12-05T19:32:47.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:47.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:47.315 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:47.354 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:47.355 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:47.613 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.94.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:32:47.614 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-65 2020-12-05T19:32:47.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:47.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:47.616 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:47.699 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:47.699 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:47.973 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.94.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:32:47.974 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-65 2020-12-05T19:32:47.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:47.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:47.975 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:48.054 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:48.055 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:48.313 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.94.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:32:48.313 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-65 2020-12-05T19:32:48.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:48.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:48.315 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:48.360 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:48.360 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:48.634 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.94.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:32:48.635 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-65 2020-12-05T19:32:48.636 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-65 zones={test-zone2, test-zone1} 2020-12-05T19:32:49.754 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:32:49.761 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-12-05T19:32:49.762 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:32:49.810 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-12-05T19:32:49.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-12-05T19:32:50.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-12-05T19:32:50.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-12-05T19:32:50.838 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-65 zones={test-zone2, test-zone1} 2020-12-05T19:32:50.956 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-65 zones={test-zone3, test-zone1} 2020-12-05T19:32:51.175 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:32:51.222 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-12-05T19:32:51.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:32:51.232 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-12-05T19:32:51.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-12-05T19:32:51.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-12-05T19:32:51.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-12-05T19:32:52.042 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-65 zones={test-zone3, test-zone1} 2020-12-05T19:32:52.043 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-65 zones={test-zone3, test-zone2} 2020-12-05T19:32:52.491 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:32:52.500 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-12-05T19:32:52.501 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:32:52.550 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-12-05T19:32:52.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-12-05T19:32:52.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-12-05T19:32:53.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-12-05T19:32:53.362 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-65 zones={test-zone3, test-zone2} 2020-12-05T19:32:53.363 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2020-12-05T19:32:53.364 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-68 2020-12-05T19:32:53.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:53.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:53.572 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:53.612 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:53.613 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:53.837 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:32:53.877 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196773.470092_653.1","last_update":"2020-12-05T19:32:53.470092Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:32:53.877 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:32:53.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:53.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:53.879 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:53.919 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:53.919 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:54.181 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196757.091315_644.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:32:37.091315Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:32:54.182 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196773.470092_653.1 target=1_1607196757.091315_644.1 2020-12-05T19:32:54.600 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:32:54.595+0000 7f6ecb7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:32:54.600 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:32:54.595+0000 7f6ee27fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:32:54.606 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:32:54.599+0000 7f6ecf7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:32:54.606 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:32:54.599+0000 7f6ecdffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:32:59.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:59.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:59.184 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:59.225 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:59.226 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:59.505 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196773.470092_653.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:32:53.470092Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:32:59.506 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:32:59.506 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:32:59.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:59.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:59.508 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:59.548 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:59.549 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:32:59.809 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196773.470092_653.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:32:53.470092Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:32:59.810 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:32:59.811 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:32:59.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:59.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:32:59.812 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:32:59.852 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:32:59.853 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:00.131 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196773.470092_653.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:32:53.470092Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:00.132 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:33:00.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:00.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:00.133 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-68 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:00.172 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:00.173 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:00.446 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.68.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.48.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-12-05T19:33:00.447 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-68 2020-12-05T19:33:00.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:00.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:00.448 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-68 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:00.494 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:00.494 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:00.773 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-kms","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-12-05T19:33:00.774 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-68 2020-12-05T19:33:01.734 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2020-12-05T19:33:01.735 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-69 2020-12-05T19:33:02.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:02.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:02.712 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:02.749 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:02.749 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:02.960 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:33:02.994 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196782.508147_659.1","last_update":"2020-12-05T19:33:02.508147Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:33:02.995 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:33:02.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:02.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:02.996 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:03.037 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:03.037 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:03.296 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196773.470092_653.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:32:53.470092Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:03.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196782.508147_659.1 target=1_1607196773.470092_653.1 2020-12-05T19:33:03.605 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:03.599+0000 7f6ec6fed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.606 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:03.599+0000 7f6ed57fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.606 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:33:03.599+0000 7f45447f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.607 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:03.596+0000 7f7f437fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.607 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:03.596+0000 7f7f42ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.608 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:33:03.599+0000 7f4559ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.609 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:03.600+0000 7f7f3f7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.609 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:03.600+0000 7f7f3c7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.610 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:33:03.603+0000 7f45417f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.610 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:33:03.603+0000 7f4541ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.610 INFO:tasks.rgw.c2.client.0.smithi023.stdout:2020-12-05T19:33:03.603+0000 7f45417f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.611 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:03.600+0000 7f7f407f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.611 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:03.603+0000 7f6ecffff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.612 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:03.603+0000 7f6ecffff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:03.612 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:03.607+0000 7f6ed4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:08.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:08.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:08.299 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:08.341 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:08.341 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:08.604 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196782.508147_659.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:02.508147Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:08.605 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:33:08.605 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:33:08.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:08.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:08.607 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:08.645 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:08.645 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:08.905 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196782.508147_659.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:02.508147Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:08.906 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:33:08.906 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:33:08.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:08.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:08.908 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:08.949 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:08.950 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:09.225 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196782.508147_659.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:02.508147Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:09.226 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:33:09.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:09.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:09.227 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:09.269 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:09.269 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:09.528 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.104.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.79.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.35.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.100.5"}]} 2020-12-05T19:33:09.528 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-69 2020-12-05T19:33:09.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:09.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:09.530 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:09.581 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:09.581 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:09.820 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.104.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.100.5"}}}] 2020-12-05T19:33:09.820 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-69 2020-12-05T19:33:09.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:09.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:09.822 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:09.864 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:09.864 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:10.116 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.104.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.79.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.35.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.100.5"}]} 2020-12-05T19:33:10.117 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-69 2020-12-05T19:33:10.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:10.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:10.118 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:10.206 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:10.207 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:10.449 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.104.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.100.5"}}}] 2020-12-05T19:33:10.449 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-69 2020-12-05T19:33:10.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:10.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:10.451 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:10.494 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:10.494 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:10.753 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.104.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.79.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.35.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.100.5"}]} 2020-12-05T19:33:10.754 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-69 2020-12-05T19:33:10.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:10.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:10.756 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:10.799 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:10.800 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:11.073 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.104.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.79.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.35.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.100.5"}}}] 2020-12-05T19:33:11.074 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-69 2020-12-05T19:33:11.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:11.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:11.075 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:11.154 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:11.154 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:11.422 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.77.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.78.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.38.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.80.5"}]} 2020-12-05T19:33:11.423 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-69 2020-12-05T19:33:11.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:11.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:11.424 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:11.467 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:11.468 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:11.752 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}}] 2020-12-05T19:33:11.752 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-69 2020-12-05T19:33:11.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:11.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:11.754 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:11.796 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:11.796 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:12.056 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.77.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.78.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.38.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.80.5"}]} 2020-12-05T19:33:12.056 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-69 2020-12-05T19:33:12.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:12.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:12.058 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:12.102 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:12.103 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:12.354 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}}] 2020-12-05T19:33:12.355 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-69 2020-12-05T19:33:12.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:12.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:12.356 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:12.437 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:12.437 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:12.691 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.102.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.91.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.40.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.92.5"}]} 2020-12-05T19:33:12.692 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-69 2020-12-05T19:33:12.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:12.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:12.693 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:12.778 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:12.778 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:13.060 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.102.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.91.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5"}}}] 2020-12-05T19:33:13.061 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-69 2020-12-05T19:33:13.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:13.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:13.062 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:13.146 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:13.146 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:13.409 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.102.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.91.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.40.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.92.5"}]} 2020-12-05T19:33:13.410 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-69 2020-12-05T19:33:13.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:13.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:13.411 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:13.751 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.102.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.91.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5"}}}] 2020-12-05T19:33:13.752 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-69 2020-12-05T19:33:13.752 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-69 zones={test-zone2, test-zone1} 2020-12-05T19:33:14.886 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:33:14.893 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:14.894 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:14.894 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:14.895 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:14.895 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:33:14.900 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:14.900 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:14.901 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:14.901 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:14.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:14.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:15.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:15.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:15.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:15.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:15.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:15.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:15.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:15.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:15.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:15.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:16.012 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-69 zones={test-zone2, test-zone1} 2020-12-05T19:33:16.012 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-69 zones={test-zone3, test-zone1} 2020-12-05T19:33:16.792 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:33:16.798 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:16.798 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:16.799 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:16.799 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:16.799 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:33:16.805 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:16.805 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:16.805 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:16.806 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:16.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:16.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:16.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:17.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:17.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:17.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:17.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:17.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:17.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:17.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:17.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:17.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:17.998 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-69 zones={test-zone3, test-zone1} 2020-12-05T19:33:17.998 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-69 zones={test-zone3, test-zone2} 2020-12-05T19:33:18.479 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:33:18.490 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:18.490 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:18.491 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:18.491 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:18.491 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:33:18.498 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:18.498 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:18.499 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:18.499 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:18.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:18.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:18.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:18.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:18.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:18.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:19.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:19.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:19.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:19.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:19.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:19.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:19.625 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-69 zones={test-zone3, test-zone2} 2020-12-05T19:33:19.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:33:19.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:33:19.627 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:33:19.661 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:19.662 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:19.942 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tazcyr-70 2020-12-05T19:33:20.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:20.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:20.888 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:20.928 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:20.929 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:21.150 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:33:21.188 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196800.684033_670.1","last_update":"2020-12-05T19:33:20.684033Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:33:21.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:33:21.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:21.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:21.190 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:21.233 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:21.234 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:21.492 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196800.684033_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:20.684033Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:21.493 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:33:21.493 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:33:21.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:21.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:21.495 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:21.534 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:21.535 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:21.804 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196782.508147_659.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:02.508147Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:21.804 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1607196800.684033_670.1 target=1_1607196782.508147_659.1 2020-12-05T19:33:21.819 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:21.811+0000 7f6eccff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.819 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:21.815+0000 7f6ed7fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.820 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:21.812+0000 7f7f3eff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.820 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:21.812+0000 7f7f42ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.821 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:21.812+0000 7f7f42ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.821 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:21.815+0000 7f6ecbff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.824 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:21.816+0000 7f7f407f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.824 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:21.816+0000 7f7f407f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.824 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:21.819+0000 7f6ecc7f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.825 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:21.819+0000 7f6ecc7f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.825 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:21.819+0000 7f6ecc7f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.825 INFO:tasks.rgw.c2.client.1.smithi023.stdout:2020-12-05T19:33:21.819+0000 7f6ecc7f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.826 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:21.820+0000 7f7f397ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:21.826 INFO:tasks.rgw.c1.client.1.smithi185.stdout:2020-12-05T19:33:21.820+0000 7f7f43fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-12-05T19:33:26.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:26.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:26.807 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:26.853 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:26.853 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:27.122 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196800.684033_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:20.684033Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:27.122 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:33:27.123 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:33:27.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:27.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:27.125 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:27.164 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:27.165 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:27.442 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196800.684033_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:20.684033Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:27.442 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:33:27.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:27.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:27.444 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:27.486 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:27.486 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:27.756 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.70.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.82.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.56.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.50.5"}]} 2020-12-05T19:33:27.756 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-70 2020-12-05T19:33:27.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:27.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:27.758 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:27.801 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:27.802 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:28.076 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}}] 2020-12-05T19:33:28.076 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=tazcyr-70 2020-12-05T19:33:28.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:28.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:28.078 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:28.120 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:28.120 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:28.383 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.70.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.82.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.56.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.50.5"}]} 2020-12-05T19:33:28.383 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-70 2020-12-05T19:33:28.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:28.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:28.385 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:28.472 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:28.473 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:28.766 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}}] 2020-12-05T19:33:28.767 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=tazcyr-70 2020-12-05T19:33:28.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:28.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:28.768 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:28.808 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:28.808 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:29.089 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.70.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.82.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.56.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.50.5"}]} 2020-12-05T19:33:29.089 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-70 2020-12-05T19:33:29.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:29.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:29.091 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket tazcyr-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:29.135 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:29.136 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:29.416 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.70.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.82.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.50.5"}}}] 2020-12-05T19:33:29.416 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=tazcyr-70 2020-12-05T19:33:29.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:29.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:29.418 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:29.498 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:29.499 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:29.768 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.55.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.81.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.55.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.50.5"}]} 2020-12-05T19:33:29.769 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-70 2020-12-05T19:33:29.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:29.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:29.771 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:29.815 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:29.815 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:30.089 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}}] 2020-12-05T19:33:30.089 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=tazcyr-70 2020-12-05T19:33:30.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:30.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:30.091 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:30.133 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:30.133 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:30.399 INFO:teuthology.orchestra.run.smithi023.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.55.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.81.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.55.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.50.5"}]} 2020-12-05T19:33:30.400 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-70 2020-12-05T19:33:30.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:30.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:30.402 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:30.447 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:30.448 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:30.732 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}}] 2020-12-05T19:33:30.733 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=tazcyr-70 2020-12-05T19:33:30.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:30.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:30.735 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:30.773 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:30.774 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:31.056 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.81.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.101.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.59.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.71.5"}]} 2020-12-05T19:33:31.057 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-70 2020-12-05T19:33:31.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:31.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:31.058 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:31.143 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:31.144 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:31.457 INFO:teuthology.orchestra.run.smithi185.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}}] 2020-12-05T19:33:31.458 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=tazcyr-70 2020-12-05T19:33:31.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:31.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:31.459 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:31.500 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:31.501 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:31.772 INFO:teuthology.orchestra.run.smithi185.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.81.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.101.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.59.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.71.5"}]} 2020-12-05T19:33:31.772 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-70 2020-12-05T19:33:31.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:31.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'tazcyr-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:31.774 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket tazcyr-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:31.813 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:32.106 INFO:teuthology.orchestra.run.smithi023.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}}] 2020-12-05T19:33:32.107 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=tazcyr-70 2020-12-05T19:33:32.108 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-70 zones={test-zone2, test-zone1} 2020-12-05T19:33:33.049 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:33:33.057 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:33.058 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:33.058 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:33.059 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:33.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:33:33.066 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:33.067 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:33.067 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:33.068 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:33.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:33.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:33.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:33.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:33.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:33.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:33.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:33.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:33.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:33.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:34.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:34.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:34.263 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-70 zones={test-zone2, test-zone1} 2020-12-05T19:33:34.264 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-70 zones={test-zone3, test-zone1} 2020-12-05T19:33:35.253 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:33:35.259 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:35.259 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:35.260 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:35.260 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:35.260 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:33:35.266 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:35.267 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:35.267 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:35.267 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:35.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:35.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:35.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:35.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:35.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:35.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:35.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:35.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:35.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:36.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:36.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:36.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:36.374 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-70 zones={test-zone3, test-zone1} 2020-12-05T19:33:36.375 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tazcyr-70 zones={test-zone3, test-zone2} 2020-12-05T19:33:36.628 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-12-05T19:33:36.637 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:36.638 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:36.639 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:36.639 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:36.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-12-05T19:33:36.648 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-12-05T19:33:36.649 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-12-05T19:33:36.649 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-12-05T19:33:36.650 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-12-05T19:33:36.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:36.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:36.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-12-05T19:33:36.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:37.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:37.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-12-05T19:33:37.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:37.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:37.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-12-05T19:33:37.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:37.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:37.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-12-05T19:33:37.926 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tazcyr-70 zones={test-zone3, test-zone2} 2020-12-05T19:33:37.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:33:37.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:33:37.928 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:33:37.959 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:37.959 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:38.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:38.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'tazcyr-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:38.253 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket tazcyr-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:38.334 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:38.334 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:38.618 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:33:38.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:38.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:38.620 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:38.701 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:38.701 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:38.982 INFO:teuthology.orchestra.run.smithi185.stdout:[{"op_id":"1#00000000001.104.5","op_tag":"e622b2e3-1a91-4dc1-b31e-43ede659e70f.6183.4806","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2020-12-05T19:33:02.518149176Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-69:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.20"}]},{"op_id":"5#00000000001.79.5","op_tag":"e622b2e3-1a91-4dc1-b31e-43ede659e70f.6183.4812","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2020-12-05T19:33:02.667160974Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-69:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.20"}]},{"op_id":"8#00000000001.35.5","op_tag":"e622b2e3-1a91-4dc1-b31e-43ede659e70f.6183.4810","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2020-12-05T19:33:02.618993652Z","ver":{"pool":9,"epoch":13},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-69:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.20"}]},{"op_id":"10#00000000001.100.5","op_tag":"e622b2e3-1a91-4dc1-b31e-43ede659e70f.6183.4808","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2020-12-05T19:33:02.566488227Z","ver":{"pool":9,"epoch":4},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"e622b2e3-1a91-4dc1-b31e-43ede659e70f:tazcyr-69:e622b2e3-1a91-4dc1-b31e-43ede659e70f.6185.20"}]}] 2020-12-05T19:33:38.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:33:38.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-12-05T19:33:38.984 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-12-05T19:33:39.060 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:39.061 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:40.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:40.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'tazcyr-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:40.637 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket tazcyr-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:40.710 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:40.710 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:40.972 INFO:teuthology.orchestra.run.smithi185.stdout:[] 2020-12-05T19:33:40.973 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2020-12-05T19:33:43.038 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-12-05T19:33:43.039 INFO:tasks.rgw_multisite_tests: 2020-12-05T19:33:43.041 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-12-05T19:33:43.041 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1447.025s 2020-12-05T19:33:43.041 INFO:tasks.rgw_multisite_tests: 2020-12-05T19:33:43.042 INFO:tasks.rgw_multisite_tests:OK 2020-12-05T19:33:43.046 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-12-05T19:33:43.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:43.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:43.048 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:43.088 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:43.089 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:43.304 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:33:43.342 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196800.684033_670.1","last_update":"2020-12-05T19:33:20.684033Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:33:43.343 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:33:43.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:43.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:43.344 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:43.654 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196800.684033_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:20.684033Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:43.655 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:33:43.655 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:33:43.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:43.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:43.658 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:43.697 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:43.697 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:43.975 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196800.684033_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:20.684033Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:43.976 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:33:43.977 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:33:43.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:43.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:43.979 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:44.019 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:44.019 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:44.296 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196800.684033_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:20.684033Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:44.297 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:33:44.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:44.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:44.298 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:44.337 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:44.338 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:44.582 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:33:44.622 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196800.684033_670.1","last_update":"2020-12-05T19:33:20.684033Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:33:44.623 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:33:44.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:44.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:44.624 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:44.665 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:44.665 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:44.939 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196800.684033_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:20.684033Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:44.939 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:33:45.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:45.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:45.501 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:45.535 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:45.535 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:45.778 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:33:45.818 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196825.490500_689.1","last_update":"2020-12-05T19:33:45.490500Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:33:45.818 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:33:45.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:45.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:45.820 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:45.861 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:45.861 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:46.146 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196800.684033_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:20.684033Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:46.147 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1607196825.490500_689.1 target=1_1607196800.684033_670.1 2020-12-05T19:33:51.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:51.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:51.149 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:51.470 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196825.490500_689.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:45.490500Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:51.471 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:33:51.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:51.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:51.707 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:51.747 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:51.747 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:51.981 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:33:52.019 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196825.490500_689.1","last_update":"2020-12-05T19:33:45.490500Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:33:52.019 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:33:52.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:52.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:52.021 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:52.060 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:52.342 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196825.490500_689.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:45.490500Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:52.344 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-12-05T19:33:52.624 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2020-12-05T19:33:52.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:52.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:52.648 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:52.699 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:52.700 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:52.939 INFO:teuthology.orchestra.run.smithi185.stderr:No --period given, using current period=99a897cd-76db-49ef-bcce-15cfc0f426ad 2020-12-05T19:33:52.967 INFO:teuthology.orchestra.run.smithi185.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1607196832.614631_693.1","last_update":"2020-12-05T19:33:52.614631Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-12-05T19:33:52.968 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-12-05T19:33:52.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:52.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:52.971 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:53.017 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:53.018 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:53.291 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196825.490500_689.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:45.490500Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:53.292 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1607196832.614631_693.1 target=1_1607196825.490500_689.1 2020-12-05T19:33:58.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:58.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:58.295 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:58.610 INFO:teuthology.orchestra.run.smithi023.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196832.614631_693.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:52.614631Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:58.610 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-12-05T19:33:58.611 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-12-05T19:33:58.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:58.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:58.613 INFO:teuthology.orchestra.run.smithi185:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:58.653 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:58.654 INFO:teuthology.orchestra.run.smithi185.stderr:ignoring --setgroup ceph since I am not root 2020-12-05T19:33:58.933 INFO:teuthology.orchestra.run.smithi185.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"99a897cd-76db-49ef-bcce-15cfc0f426ad","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1607195426.378999_125.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-12-05T19:10:26.378999Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1607196832.614631_693.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-12-05T19:33:52.614631Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-12-05T19:33:58.934 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-12-05T19:33:58.934 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-12-05T19:33:58.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:58.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-12-05T19:33:58.936 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-12-05T19:33:58.977 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setuser ceph since I am not root 2020-12-05T19:33:58.978 INFO:teuthology.orchestra.run.smithi023.stderr:ignoring --setgroup ceph since I am not root 2020-12-06T06:51:32.855 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-12-06T06:51:32.899 DEBUG:teuthology.task.console_log:Killing console logger for smithi185 2020-12-06T06:51:32.899 DEBUG:teuthology.task.console_log:Killing console logger for smithi023 2020-12-06T06:51:32.900 DEBUG:teuthology.task.console_log:Killing console logger for smithi185 2020-12-06T06:51:32.900 DEBUG:teuthology.task.console_log:Killing console logger for smithi023 2020-12-06T06:51:32.901 DEBUG:teuthology.exit:Finished running handlers